[jira] [Reopened] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-12762:
No fix version for 0.98 was set on this issue but it was committed there.
TestShell is failing on 0.98 since this change, please see
https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/728 and
https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/729
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12783) Create efficient RegionLocator implementation
Solomon Duskis created HBASE-12783: -- Summary: Create efficient RegionLocator implementation Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12784) TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes hangs
Ted Yu created HBASE-12784:
--
Summary:
TestFastFailWithoutTestUtil#testPreemptiveFastFailException50Times sometimes
hangs
Key: HBASE-12784
URL: https://issues.apache.org/jira/browse/HBASE-12784
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
I was running test suite against hadoop 2.7.0-SNAPSHOT and saw this:
{code}
testPreemptiveFastFailException50Times(org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil)
Time elapsed: 120.013 sec ERROR!
java.lang.Exception: test timed out after 12 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException(TestFastFailWithoutTestUtil.java:450)
at
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil.testPreemptiveFastFailException50Times(TestFastFailWithoutTestUtil.java:338)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
Ted Yu created HBASE-12785: -- Summary: Use FutureTask to timeout the attempt to get the lock for hbck Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Priority: Minor In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12762) Region with no hfiles will have the highest locality cost in LocalityCostFunction
[
https://issues.apache.org/jira/browse/HBASE-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-12762.
Resolution: Fixed
Region with no hfiles will have the highest locality cost in
LocalityCostFunction
-
Key: HBASE-12762
URL: https://issues.apache.org/jira/browse/HBASE-12762
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12762-trunk.patch
The locality cost of region will be computed in LocalityCostFunction.cost as:
{code}
double cost() {
...
int index = -1;
for (int j = 0; j regionLocations.length; j++) {
if (regionLocations[j] = 0 regionLocations[j] == serverIndex) {
index = j;
break;
}
}
if (index 0) {
cost += 1; // == region with no hfiles will have the highest cost
} else {
cost += (double) index / (double) regionLocations.length;
}
...
}
{code}
The region with no hfiles(such as empty region) will have the highest cost
which represents the worst case that region located in the server with no
locality for hfiles. However, this might be the best case because there are
no hlogs for the region. Although the absolute cost value won't affect the
balance process, will it be more reasonable to have zero cost for such
regions? such as:
{code}
...
if (index 0) {
if (regionLocation.length 0) { // == only consider regions with
hfiles
cost += 1;
}
} else {
cost += (double) index / (double) regionLocations.length;
}
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: A face-lift for 1.0
Not sure if this is something to address in this thread or if I should file a separate JIRA, but for some reason, accessing the site over https gets rid of the pretty CSS. Seems to happen in Chrome and Firefox. Anyone have a clue why? -Dima On Wed, Dec 17, 2014 at 10:53 AM, Nick Dimiduk ndimi...@gmail.com wrote: Tweaking the layout of the website goes one of two ways. Either there's a configuration point exposed from the maven skin or there isn't. If there is, it's simply editing the appropriate custom tag in site.xml. If not, we'll need to bring back the velocity site.vm template. Configurations are documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/config.html Tweaking visuals is now based on the theme. We can either choose another bootswatch theme, or write our own. That stuff is documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/themes/ Right now the site is separate from the book. In theory we could move book content over to become site content, in which case it would be managed by this same plugin. Short of that, we get into applying style via the docbook plugin. I've looked into this a bit; it's not nearly as easy as applying reflow-maven-skin. The current leads I have for this are https://github.com/lcahlander/docbook and https://github.com/pmartin/phing-documentation-docbook/commit/8d8f0b81951c4e07420083b204599a6b9ef00537 . Honestly though, if we're looking at moving away from docbook, I think we should consider a more modern toolset, Sphinx-doc, for example: http://sphinx-doc.org On Tue, Dec 16, 2014 at 8:21 PM, Misty Stanley-Jones mstanleyjo...@cloudera.com wrote: Absolutely fantastic!!! Got pointers to how tweaking the website would work? In the new year I hope to really get started on a restructure of the Ref Guide. I've thought about asking to move to AsciiDOC or MarkDown but I'd like to do a proof-of-concept and see if it really gets us anything we don't already have with Docbook. It's less of a potential value since we are not hosted on Github, which has great integration with AsciiDOC and MarkDown. I'm very interested in improving the look and feel of the doc. I think a lot of it will be structural -- reorganizing and re-chunking content. In the meantime, theming the Docbook is not as hard as you might think. Have a look at the XSLT files. (I know, famous last words). On Wed, Dec 17, 2014 at 1:21 PM, Nick Dimiduk ndimi...@gmail.com wrote: For those not following closely in JIRAland, I've two changes that are ready to go. First, there's transition over to a bootstrap-based layout for our home page. This work is done in HBASE-12688. This improves the looks of things and will make it pretty easy to update our look-and-feel in the future. Second, I fixed a build problem where the online book wasn't getting its intended styling. This is done in HBASE-12687. I'd like to update the book to look the same as the front-page, but getting styles into docbook is proving non-trivial. Anyway, both of these changes are pushed to master and we're ready for a site re-build with the new look. I'll make that happen tomorrow morning unless I hear actionable objections. You can get a preview of things to come over one http://people.apache.org/~ndimiduk/site/ Thanks, Nick On Fri, Dec 5, 2014 at 12:29 PM, Stack st...@duboce.net wrote: Thanks. I've been having a dialog w/ your crew that has been going on a while... was wondering if any progress. Grand, St.Ack On Fri, Dec 5, 2014 at 11:55 AM, Ryan Rawson ryano...@gmail.com wrote: I'm checking to see if our marketing and web team can help. The primary requirement is going to be ditching the mvn:site from the front page. reskinning it might not be so easy. -ryan On Thu, Dec 4, 2014 at 9:52 AM, lars hofhansl la...@apache.org wrote: +1 I just came across one of the various HBase vs. Cassandra articles, and one of the main tenants of the articles was how much better the Cassandra documentation was.Thank god we have Misty now. :) (not sure how much just a skin would help, but it surely won't hurt) -- Lars From: Nick Dimiduk ndimi...@gmail.com To: hbase-dev dev@hbase.apache.org Sent: Tuesday, December 2, 2014 9:46 AM Subject: A face-lift for 1.0 Heya, In mind of the new release, I was thinking we should clean up our act a little bit in regard to hbase.apache.org and our book. Just because the project started in 2007 doesn't mean we need a site that looks like it's from 2007. Phoenix's site looks great in this regard. For the home page, I was thinking of converting it over to bootstrap [0] so that it'll be easier to pick up theme, either on of our own
Re: A face-lift for 1.0
I had the same problem. We link to CSS which is http instead of forking and serving it in the site (I believe this is the default practice). My chrome extension refuses to connect to http links causing rendering problems. Enis On Tue, Dec 30, 2014 at 11:00 AM, Dima Spivak dspi...@cloudera.com wrote: Not sure if this is something to address in this thread or if I should file a separate JIRA, but for some reason, accessing the site over https gets rid of the pretty CSS. Seems to happen in Chrome and Firefox. Anyone have a clue why? -Dima On Wed, Dec 17, 2014 at 10:53 AM, Nick Dimiduk ndimi...@gmail.com wrote: Tweaking the layout of the website goes one of two ways. Either there's a configuration point exposed from the maven skin or there isn't. If there is, it's simply editing the appropriate custom tag in site.xml. If not, we'll need to bring back the velocity site.vm template. Configurations are documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/config.html Tweaking visuals is now based on the theme. We can either choose another bootswatch theme, or write our own. That stuff is documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/themes/ Right now the site is separate from the book. In theory we could move book content over to become site content, in which case it would be managed by this same plugin. Short of that, we get into applying style via the docbook plugin. I've looked into this a bit; it's not nearly as easy as applying reflow-maven-skin. The current leads I have for this are https://github.com/lcahlander/docbook and https://github.com/pmartin/phing-documentation-docbook/commit/8d8f0b81951c4e07420083b204599a6b9ef00537 . Honestly though, if we're looking at moving away from docbook, I think we should consider a more modern toolset, Sphinx-doc, for example: http://sphinx-doc.org On Tue, Dec 16, 2014 at 8:21 PM, Misty Stanley-Jones mstanleyjo...@cloudera.com wrote: Absolutely fantastic!!! Got pointers to how tweaking the website would work? In the new year I hope to really get started on a restructure of the Ref Guide. I've thought about asking to move to AsciiDOC or MarkDown but I'd like to do a proof-of-concept and see if it really gets us anything we don't already have with Docbook. It's less of a potential value since we are not hosted on Github, which has great integration with AsciiDOC and MarkDown. I'm very interested in improving the look and feel of the doc. I think a lot of it will be structural -- reorganizing and re-chunking content. In the meantime, theming the Docbook is not as hard as you might think. Have a look at the XSLT files. (I know, famous last words). On Wed, Dec 17, 2014 at 1:21 PM, Nick Dimiduk ndimi...@gmail.com wrote: For those not following closely in JIRAland, I've two changes that are ready to go. First, there's transition over to a bootstrap-based layout for our home page. This work is done in HBASE-12688. This improves the looks of things and will make it pretty easy to update our look-and-feel in the future. Second, I fixed a build problem where the online book wasn't getting its intended styling. This is done in HBASE-12687. I'd like to update the book to look the same as the front-page, but getting styles into docbook is proving non-trivial. Anyway, both of these changes are pushed to master and we're ready for a site re-build with the new look. I'll make that happen tomorrow morning unless I hear actionable objections. You can get a preview of things to come over one http://people.apache.org/~ndimiduk/site/ Thanks, Nick On Fri, Dec 5, 2014 at 12:29 PM, Stack st...@duboce.net wrote: Thanks. I've been having a dialog w/ your crew that has been going on a while... was wondering if any progress. Grand, St.Ack On Fri, Dec 5, 2014 at 11:55 AM, Ryan Rawson ryano...@gmail.com wrote: I'm checking to see if our marketing and web team can help. The primary requirement is going to be ditching the mvn:site from the front page. reskinning it might not be so easy. -ryan On Thu, Dec 4, 2014 at 9:52 AM, lars hofhansl la...@apache.org wrote: +1 I just came across one of the various HBase vs. Cassandra articles, and one of the main tenants of the articles was how much better the Cassandra documentation was.Thank god we have Misty now. :) (not sure how much just a skin would help, but it surely won't hurt) -- Lars From: Nick Dimiduk ndimi...@gmail.com To: hbase-dev dev@hbase.apache.org Sent: Tuesday, December 2, 2014 9:46 AM Subject: A face-lift for 1.0 Heya, In mind of the new release, I was
Re: A face-lift for 1.0
Ooh, good call, guys. Yeah, looks like all we'd need to do is move bootstrap.min.css and bootstrap-responsive.min.css onto our server and link to it relatively instead of with absolute links it has now. Seeing as how both of those CSS files are Apache License, should be easy. -Dima On Tue, Dec 30, 2014 at 11:54 AM, Elliott Clark ecl...@apache.org wrote: Yeah the js and css are all loaded though a cdn. It should be pretty easy to change the urls to be protocol relative and then things should work. On Tue, Dec 30, 2014 at 11:17 AM, Enis Söztutar enis@gmail.com wrote: I had the same problem. We link to CSS which is http instead of forking and serving it in the site (I believe this is the default practice). My chrome extension refuses to connect to http links causing rendering problems. Enis On Tue, Dec 30, 2014 at 11:00 AM, Dima Spivak dspi...@cloudera.com wrote: Not sure if this is something to address in this thread or if I should file a separate JIRA, but for some reason, accessing the site over https gets rid of the pretty CSS. Seems to happen in Chrome and Firefox. Anyone have a clue why? -Dima On Wed, Dec 17, 2014 at 10:53 AM, Nick Dimiduk ndimi...@gmail.com wrote: Tweaking the layout of the website goes one of two ways. Either there's a configuration point exposed from the maven skin or there isn't. If there is, it's simply editing the appropriate custom tag in site.xml. If not, we'll need to bring back the velocity site.vm template. Configurations are documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/config.html Tweaking visuals is now based on the theme. We can either choose another bootswatch theme, or write our own. That stuff is documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/themes/ Right now the site is separate from the book. In theory we could move book content over to become site content, in which case it would be managed by this same plugin. Short of that, we get into applying style via the docbook plugin. I've looked into this a bit; it's not nearly as easy as applying reflow-maven-skin. The current leads I have for this are https://github.com/lcahlander/docbook and https://github.com/pmartin/phing-documentation-docbook/commit/8d8f0b81951c4e07420083b204599a6b9ef00537 . Honestly though, if we're looking at moving away from docbook, I think we should consider a more modern toolset, Sphinx-doc, for example: http://sphinx-doc.org On Tue, Dec 16, 2014 at 8:21 PM, Misty Stanley-Jones mstanleyjo...@cloudera.com wrote: Absolutely fantastic!!! Got pointers to how tweaking the website would work? In the new year I hope to really get started on a restructure of the Ref Guide. I've thought about asking to move to AsciiDOC or MarkDown but I'd like to do a proof-of-concept and see if it really gets us anything we don't already have with Docbook. It's less of a potential value since we are not hosted on Github, which has great integration with AsciiDOC and MarkDown. I'm very interested in improving the look and feel of the doc. I think a lot of it will be structural -- reorganizing and re-chunking content. In the meantime, theming the Docbook is not as hard as you might think. Have a look at the XSLT files. (I know, famous last words). On Wed, Dec 17, 2014 at 1:21 PM, Nick Dimiduk ndimi...@gmail.com wrote: For those not following closely in JIRAland, I've two changes that are ready to go. First, there's transition over to a bootstrap-based layout for our home page. This work is done in HBASE-12688. This improves the looks of things and will make it pretty easy to update our look-and-feel in the future. Second, I fixed a build problem where the online book wasn't getting its intended styling. This is done in HBASE-12687. I'd like to update the book to look the same as the front-page, but getting styles into docbook is proving non-trivial. Anyway, both of these changes are pushed to master and we're ready for a site re-build with the new look. I'll make that happen tomorrow morning unless I hear actionable objections. You can get a preview of things to come over one http://people.apache.org/~ndimiduk/site/ Thanks, Nick On Fri, Dec 5, 2014 at 12:29 PM, Stack st...@duboce.net wrote: Thanks. I've been having a dialog w/ your crew that has been going on a while... was wondering if any progress. Grand, St.Ack On Fri, Dec 5, 2014 at 11:55 AM, Ryan Rawson ryano...@gmail.com wrote: I'm checking to
Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise.
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise.
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise.
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
It would be a binary compatibility break unless we detect by reflection that it's an older factory missing the new 'create' method and therefore call the old one. We could add that. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? On Dec 30, 2014, at 3:23 PM, James Taylor jamestay...@apache.org wrote: Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise.
[jira] [Reopened] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-12270: This broke the 0.98 build [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java:[224,8] cannot find symbol symbol : constructor CacheConfig(org.apache.hadoop.hbase.io.hfile.BlockCache,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) location: class org.apache.hadoop.abase.io.hfile.CacheConfig [INFO] 1 error A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
bq. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? Good question. Do you have an opinion? We have a compatibility check that we do on first connection to a cluster. Perhaps we can add a check of Phoenix server version vs HBase server version to detect a breakage scenario? In this case, we'd require the server-side Phoenix version to be bumped up (maybe do this in 4.4?). We can doc it as well, but it's been my experience that folks just don't read this. So perhaps have the reflection in place in HBase long enough for us to get 4.4 out? Thanks for asking! James On Tue, Dec 30, 2014 at 3:36 PM, Andrew Purtell andrew.purt...@gmail.com wrote: It would be a binary compatibility break unless we detect by reflection that it's an older factory missing the new 'create' method and therefore call the old one. We could add that. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? On Dec 30, 2014, at 3:23 PM, James Taylor jamestay...@apache.org wrote: Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise.
[jira] [Created] (HBASE-12787) Backport HBASE-12028 (Abort the RegionServer when it's handler threads die) to 0.98
Andrew Purtell created HBASE-12787: -- Summary: Backport HBASE-12028 (Abort the RegionServer when it's handler threads die) to 0.98 Key: HBASE-12787 URL: https://issues.apache.org/jira/browse/HBASE-12787 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Backport HBASE-12028 (Abort the RegionServer when it's handler threads die) to 0.98. There are two 0.98-specific changes that should be addressed: # The default configuration should preserve current behavior # The interface change to RpcSchedulerFactory should be binary compatible for Phoenix if possible, and we can do this by detecting with reflection if an older implementation of the interface is missing the new 'create' method and by invoking the older deprecated 'create' method instead. We can cache the decision after making it once. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
Thanks for the feedback James. I filed HBASE-12787 in response. On Tue, Dec 30, 2014 at 3:49 PM, James Taylor jamestay...@apache.org wrote: bq. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? Good question. Do you have an opinion? We have a compatibility check that we do on first connection to a cluster. Perhaps we can add a check of Phoenix server version vs HBase server version to detect a breakage scenario? In this case, we'd require the server-side Phoenix version to be bumped up (maybe do this in 4.4?). We can doc it as well, but it's been my experience that folks just don't read this. So perhaps have the reflection in place in HBase long enough for us to get 4.4 out? Thanks for asking! James On Tue, Dec 30, 2014 at 3:36 PM, Andrew Purtell andrew.purt...@gmail.com wrote: It would be a binary compatibility break unless we detect by reflection that it's an older factory missing the new 'create' method and therefore call the old one. We could add that. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? On Dec 30, 2014, at 3:23 PM, James Taylor jamestay...@apache.org wrote: Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
If we would like to port this change to 0.98, another option is not changing RpcSchedulerFactory, but add a SET method for RpcScheduler that set the Abortable afterwards. Thus there will be no backward compatible issue. New code need to know to call the SET method. Old code will have a null abortable. Alicia On Tue, Dec 30, 2014 at 3:49 PM, James Taylor jamestay...@apache.org wrote: bq. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? Good question. Do you have an opinion? We have a compatibility check that we do on first connection to a cluster. Perhaps we can add a check of Phoenix server version vs HBase server version to detect a breakage scenario? In this case, we'd require the server-side Phoenix version to be bumped up (maybe do this in 4.4?). We can doc it as well, but it's been my experience that folks just don't read this. So perhaps have the reflection in place in HBase long enough for us to get 4.4 out? Thanks for asking! James On Tue, Dec 30, 2014 at 3:36 PM, Andrew Purtell andrew.purt...@gmail.com wrote: It would be a binary compatibility break unless we detect by reflection that it's an older factory missing the new 'create' method and therefore call the old one. We could add that. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? On Dec 30, 2014, at 3:23 PM, James Taylor jamestay...@apache.org wrote: Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HBASE-12788) Promote Abortable to LimitedPrivate
Andrew Purtell created HBASE-12788: -- Summary: Promote Abortable to LimitedPrivate Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12789) TestCacheOnWrite#testStoreFileCacheOnWrite sometimes fails with HFileBlock cached assertion failure
Ted Yu created HBASE-12789:
--
Summary: TestCacheOnWrite#testStoreFileCacheOnWrite sometimes
fails with HFileBlock cached assertion failure
Key: HBASE-12789
URL: https://issues.apache.org/jira/browse/HBASE-12789
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From
https://builds.apache.org/job/HBase-TRUNK/5980/testReport/junit/org.apache.hadoop.hbase.io.hfile/TestCacheOnWrite/testStoreFileCacheOnWrite_106_/
:
{code}
java.lang.AssertionError: shouldBeCached: false
isCached: true
Test description: [cacheOnWrite=INDEX_BLOCKS, compress=GZ,
encoderType=BLOCK_ENCODING_EVERYWHERE, cacheCompressedData=false]
block: HFileBlock [ fileOffset=12046 headerSize()=33 blockType=ENCODED_DATA
onDiskSizeWithoutHeader=1325 uncompressedSizeWithoutHeader=1591
prevBlockOffset=10457 isUseHBaseChecksum()=true checksumType=NULL
bytesPerChecksum=512 onDiskDataSizeWithHeader=1346
getOnDiskSizeWithHeader()=1358 totalChecksumBytes()=12 isUnpacked()=true buf=[
java.nio.HeapByteBuffer[pos=0 lim=1636 cap=1669] ]
dataBeginsWith=\x00\x02\x00\x00\x08N.\x1D\x00\x00 a
fileContext=HFileContext [ usesHBaseChecksum=true checksumType=NULL
bytesPerChecksum=512 blocksize=65536 encoding=NONE includesMvcc=true
includesTags=true compressAlgo=NONE compressTags=false cryptoContext=[
cipher=NONE keyHash=NONE ] ] ]
encodingInCache: PREFIX
blockCacheKey: bfdf56eb18464371a17ed05500e189d6_12046
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.readStoreFile(TestCacheOnWrite.java:285)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWriteInternals(TestCacheOnWrite.java:244)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:239)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12789) TestCacheOnWrite#testStoreFileCacheOnWrite sometimes fails with HFileBlock cached assertion failure
[
https://issues.apache.org/jira/browse/HBASE-12789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-12789.
---
Resolution: Invalid
Copy/paste of our build box output with no input.
TestCacheOnWrite#testStoreFileCacheOnWrite sometimes fails with HFileBlock
cached assertion failure
---
Key: HBASE-12789
URL: https://issues.apache.org/jira/browse/HBASE-12789
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From
https://builds.apache.org/job/HBase-TRUNK/5980/testReport/junit/org.apache.hadoop.hbase.io.hfile/TestCacheOnWrite/testStoreFileCacheOnWrite_106_/
:
{code}
java.lang.AssertionError: shouldBeCached: false
isCached: true
Test description: [cacheOnWrite=INDEX_BLOCKS, compress=GZ,
encoderType=BLOCK_ENCODING_EVERYWHERE, cacheCompressedData=false]
block: HFileBlock [ fileOffset=12046 headerSize()=33 blockType=ENCODED_DATA
onDiskSizeWithoutHeader=1325 uncompressedSizeWithoutHeader=1591
prevBlockOffset=10457 isUseHBaseChecksum()=true checksumType=NULL
bytesPerChecksum=512 onDiskDataSizeWithHeader=1346
getOnDiskSizeWithHeader()=1358 totalChecksumBytes()=12 isUnpacked()=true
buf=[ java.nio.HeapByteBuffer[pos=0 lim=1636 cap=1669] ]
dataBeginsWith=\x00\x02\x00\x00\x08N.\x1D\x00\x00 a
fileContext=HFileContext [ usesHBaseChecksum=true checksumType=NULL
bytesPerChecksum=512 blocksize=65536 encoding=NONE includesMvcc=true
includesTags=true compressAlgo=NONE compressTags=false cryptoContext=[
cipher=NONE keyHash=NONE ] ] ]
encodingInCache: PREFIX
blockCacheKey: bfdf56eb18464371a17ed05500e189d6_12046
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.readStoreFile(TestCacheOnWrite.java:285)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWriteInternals(TestCacheOnWrite.java:244)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testStoreFileCacheOnWrite(TestCacheOnWrite.java:239)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: A face-lift for 1.0
Any chance of fixing this by way of a contribution back to the maven-skin maintainers? We can avoid bringing back the site.vm if we leave this in their hands. On Tue, Dec 30, 2014 at 12:11 PM, Dima Spivak dspi...@cloudera.com wrote: Ooh, good call, guys. Yeah, looks like all we'd need to do is move bootstrap.min.css and bootstrap-responsive.min.css onto our server and link to it relatively instead of with absolute links it has now. Seeing as how both of those CSS files are Apache License, should be easy. -Dima On Tue, Dec 30, 2014 at 11:54 AM, Elliott Clark ecl...@apache.org wrote: Yeah the js and css are all loaded though a cdn. It should be pretty easy to change the urls to be protocol relative and then things should work. On Tue, Dec 30, 2014 at 11:17 AM, Enis Söztutar enis@gmail.com wrote: I had the same problem. We link to CSS which is http instead of forking and serving it in the site (I believe this is the default practice). My chrome extension refuses to connect to http links causing rendering problems. Enis On Tue, Dec 30, 2014 at 11:00 AM, Dima Spivak dspi...@cloudera.com wrote: Not sure if this is something to address in this thread or if I should file a separate JIRA, but for some reason, accessing the site over https gets rid of the pretty CSS. Seems to happen in Chrome and Firefox. Anyone have a clue why? -Dima On Wed, Dec 17, 2014 at 10:53 AM, Nick Dimiduk ndimi...@gmail.com wrote: Tweaking the layout of the website goes one of two ways. Either there's a configuration point exposed from the maven skin or there isn't. If there is, it's simply editing the appropriate custom tag in site.xml. If not, we'll need to bring back the velocity site.vm template. Configurations are documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/config.html Tweaking visuals is now based on the theme. We can either choose another bootswatch theme, or write our own. That stuff is documented here: http://andriusvelykis.github.io/reflow-maven-skin/skin/themes/ Right now the site is separate from the book. In theory we could move book content over to become site content, in which case it would be managed by this same plugin. Short of that, we get into applying style via the docbook plugin. I've looked into this a bit; it's not nearly as easy as applying reflow-maven-skin. The current leads I have for this are https://github.com/lcahlander/docbook and https://github.com/pmartin/phing-documentation-docbook/commit/8d8f0b81951c4e07420083b204599a6b9ef00537 . Honestly though, if we're looking at moving away from docbook, I think we should consider a more modern toolset, Sphinx-doc, for example: http://sphinx-doc.org On Tue, Dec 16, 2014 at 8:21 PM, Misty Stanley-Jones mstanleyjo...@cloudera.com wrote: Absolutely fantastic!!! Got pointers to how tweaking the website would work? In the new year I hope to really get started on a restructure of the Ref Guide. I've thought about asking to move to AsciiDOC or MarkDown but I'd like to do a proof-of-concept and see if it really gets us anything we don't already have with Docbook. It's less of a potential value since we are not hosted on Github, which has great integration with AsciiDOC and MarkDown. I'm very interested in improving the look and feel of the doc. I think a lot of it will be structural -- reorganizing and re-chunking content. In the meantime, theming the Docbook is not as hard as you might think. Have a look at the XSLT files. (I know, famous last words). On Wed, Dec 17, 2014 at 1:21 PM, Nick Dimiduk ndimi...@gmail.com wrote: For those not following closely in JIRAland, I've two changes that are ready to go. First, there's transition over to a bootstrap-based layout for our home page. This work is done in HBASE-12688. This improves the looks of things and will make it pretty easy to update our look-and-feel in the future. Second, I fixed a build problem where the online book wasn't getting its intended styling. This is done in HBASE-12687. I'd like to update the book to look the same as the front-page, but getting styles into docbook is proving non-trivial. Anyway, both of these changes are pushed to master and we're ready for a site re-build with the new look. I'll make that happen tomorrow morning unless I hear actionable objections. You can get a preview of things to come over one http://people.apache.org/~ndimiduk/site/
[jira] [Resolved] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-12270. Resolution: Fixed Something weird happened with TestReplicationEndpoint in the build where the addendum went in but it could be spurious and the compile problem is definitely fixed. A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Considering a RpcSchedulerFactory change in 0.98 for HBASE-12028
Thanks, Andrew. I filed PHOENIX-1569 as well. On Tue, Dec 30, 2014 at 4:34 PM, Andrew Purtell apurt...@apache.org wrote: Thanks for the feedback James. I filed HBASE-12787 in response. On Tue, Dec 30, 2014 at 3:49 PM, James Taylor jamestay...@apache.org wrote: bq. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? Good question. Do you have an opinion? We have a compatibility check that we do on first connection to a cluster. Perhaps we can add a check of Phoenix server version vs HBase server version to detect a breakage scenario? In this case, we'd require the server-side Phoenix version to be bumped up (maybe do this in 4.4?). We can doc it as well, but it's been my experience that folks just don't read this. So perhaps have the reflection in place in HBase long enough for us to get 4.4 out? Thanks for asking! James On Tue, Dec 30, 2014 at 3:36 PM, Andrew Purtell andrew.purt...@gmail.com wrote: It would be a binary compatibility break unless we detect by reflection that it's an older factory missing the new 'create' method and therefore call the old one. We could add that. How many versions of HBase = 0.98.10 do you think would need to be binary compatible with 4.2.2? On Dec 30, 2014, at 3:23 PM, James Taylor jamestay...@apache.org wrote: Would our 4.2.2 binaries continue to work with releases of HBase containing this change? On Tue, Dec 30, 2014 at 3:14 PM, Enis Söztutar enis@gmail.com wrote: Thanks Andrew, Once HBASE-12028 is committed it should be easy enough to make the changes in Phoenix to be able to compile with HBase versions pre or post HBASE-12028. But we need a PHOENIX issue for that. We should also make Abortable a LimitedPrivate it seems. Enis On Tue, Dec 30, 2014 at 2:49 PM, Andrew Purtell andrew.purt...@gmail.com wrote: Hi Phoenix, Please see https://issues.apache.org/jira/browse/HBASE-12028 The proposed change if committed into 0.98 branch would introduce a new 'create' method into the RpcSchedulerFactory interface that receives an Abortable as an additional parameter. Thus, the factory can pass this on to schedulers and workers and if something terrible happens in or to a RPC handler they can trigger a server abort. Due to a design oversight we don't otherwise have this capability. In my opinion it is important to fix this oversight. (Phoenix can also potentially make use of the Abortable for fatal issues involving indexes.) Otherwise RPC handlers can silently terminate upon receiving an unhandled throwable, potentially leaving behind bad state, certainly impacting performance and availability. However because RpcSchedulerFactory is an interface any implementor will not compile after this change, until updated. HBase could include this change in the next 0.98 release or not. Please advise. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12790) Support fairness across parallelized scans
James Taylor created HBASE-12790: Summary: Support fairness across parallelized scans Key: HBASE-12790 URL: https://issues.apache.org/jira/browse/HBASE-12790 Project: HBase Issue Type: Bug Reporter: James Taylor Some HBase clients parallelize the execution of a scan to reduce latency in getting back results. This can lead to starvation with a loaded cluster and interleaved scans, since the RPC queue will be ordered and processed on a FIFO basis. For example, if there are two clients, A B that submit largish scans at the same time. Say each scan is broken down into 100 scans by the client (broken down into equal depth chunks along the row key), and the 100 scans of client A are queued first, followed immediately by the 100 scans of client B. In this case, client B will be starved out of getting any results back until the scans for client A complete. One solution to this is to use the attached AbstractRoundRobinQueue instead of the standard FIFO queue. The queue to be used could be (maybe it already is) configurable based on a new config parameter. Using this queue would require the client to have the same identifier for all of the 100 parallel scans that represent a single logical scan from the clients point of view. With this information, the round robin queue would pick off a task from the queue in a round robin fashion (instead of a strictly FIFO manner) to prevent starvation over interleaved parallelized scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
Rajeshbabu Chintaguntla created HBASE-12791:
---
Summary: HBase does not attempt to clean up an aborted split when
the regionserver shutting down
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
HBase not cleaning the daughter region directories from HDFS if region server
shut down after creating the daughter region directories during the split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions
flushes
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Updates disabled for region
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:41,516 INFO
[StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1]
regionserver.HStore: Closed f
2014-12-31 09:05:41,518 INFO [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closed t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for
table t dd9731ee43b104da565257ca1539aa8c
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for
table t 2e40a44511c0e187d357d651f13a1dab
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
2014-12-31 09:06:30,465 INFO [Thread-8] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
{code}
- Skipping rollback if RS stopped or stopping so we end up in dirty daughter
regions in HDFS.
{code}
2014-12-31 09:07:49,547 INFO [regionserver60020-splits-1419996941385]
regionserver.SplitRequest: Skip rollback/cleanup of failed split of
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
java.io.InterruptedIOException: Interrupted after 0 tries on 350
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
{code}
Because of this hbck always showing inconsistencies.
{code}
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any region
server
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any region
server
{code}
If we try to repair then we end up in overlap regions in hbase:meta.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12792) [backport] HBASE-5835: Catch and handle NotServingRegionException when close region attempt fails
Esteban Gutierrez created HBASE-12792: - Summary: [backport] HBASE-5835: Catch and handle NotServingRegionException when close region attempt fails Key: HBASE-12792 URL: https://issues.apache.org/jira/browse/HBASE-12792 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.94.26 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Trivial Fix For: 0.94.27 This one is around in 0.94 and its a low hanging fruit when we get a NotServerRegionException if the region is not found when we attempt to close. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
Esteban Gutierrez created HBASE-12793: - Summary: [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires Key: HBASE-12793 URL: https://issues.apache.org/jira/browse/HBASE-12793 Project: HBase Issue Type: Bug Components: hbck Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor This is subtask on HBASE-12131 in order to handle gracefully network partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
