[jira] [Commented] (HBASE-6576) HBaseAdmin.createTable should wait until the table is enabled
[
https://issues.apache.org/jira/browse/HBASE-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435779#comment-13435779
]
Lars Hofhansl commented on HBASE-6576:
--
@Ted: are planning integrate into 0.94 too? (and 0.92)?
HBaseAdmin.createTable should wait until the table is enabled
-
Key: HBASE-6576
URL: https://issues.apache.org/jira/browse/HBASE-6576
Project: HBase
Issue Type: Bug
Components: client, test
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: HBASE-6576-92.patch, HBASE-6576-94.patch,
HBASE-6576-trunk.patch
The function:
{code}
public void createTable(final HTableDescriptor desc, byte [][] splitKeys)
{code}
in HBaseAdmin is synchronous and returns once all the regions of the table
are online, but does not wait for the table to be enabled, which is the last
step of table creation (see CreateTableHandler).
This is confusing and leads to racy code because users do not realize that
this is the case. For example, I saw the following test failure in 0.92 when
I ran:
mvn test
-Dtest=org.apache.hadoop.hbase.client.TestAdmin#testEnableDisableAddColumnDeleteColumn
{code}
Error Message
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Stacktrace
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
{code}
The issue is that code will create and table and immediately disable it in
order to do some testing, for example, to test an operation that only works
when the table is disabled. If the table has not been enabled yet, they will
get back a TableNotEnabledException.
The specific test above was fixed in HBASE-5206, but other examples exist in
the code, for example the following:
{code}
hbase org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat newtable asdf14
{code}
The code in question is:
{code}
byte[] tname = args[1].getBytes();
HTable table = util.createTable(tname, FAMILIES);
HBaseAdmin admin = new HBaseAdmin(conf);
admin.disableTable(tname);
{code}
It would be better if createTable just waited until the table was enabled, or
threw a TableNotEnabledException if it exhausted the configured number of
retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435780#comment-13435780 ] Lars Hofhansl commented on HBASE-6580: -- I can even see replacing the current HTablePool with this. It is simpler, should be faster, and definitely more lightweight. New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor - Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Fix For: 0.96.0, 0.94.2 Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6538) Remove copy_table.rb script
[ https://issues.apache.org/jira/browse/HBASE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435786#comment-13435786 ] Andrew Wang commented on HBASE-6538: Pretty sure findbugs and javac are unrelated, since this is a trivial patch. Remove copy_table.rb script --- Key: HBASE-6538 URL: https://issues.apache.org/jira/browse/HBASE-6538 Project: HBase Issue Type: Task Components: scripts Affects Versions: 0.96.0 Reporter: David S. Wang Assignee: Andrew Wang Priority: Minor Labels: noob Attachments: hbase-6583-1.patch Remove copy_table.rb script as per mailing list discussion. It hasn't been maintained in a while and does not run against any recent HBase release. There is also an MR job to do the same thing that does work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435805#comment-13435805
]
chunhui shen commented on HBASE-6587:
-
@ram
{code}
2012-08-14 20:42:54,367 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Unable to determine a plan to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
{code}
After the above log, TimeoutMonitor set allRegionServersOffline true
{code}2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available {code}
At the 2012-08-14 20:44:31, one server is onlined now, and region
277b9b6df6de2b9be1353b4fa25f4222 is sucessfully assigned.
However, at that time TimeoutMonitor, in th chore(), it would act on time out
because the if block {
code}if (this.allRegionServersOffline !allRSsOffline){code} return true;
So we see the following log
{code}2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OPENING for too long, reassigning
region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
The region is assigned at the time 2012-08-14 20:44:31, but is timed out by
TimeoutMonitor at the time 2012-08-14 20:44:32.
It cause the collision by two assign thread,
And the result is that the region is onlined after 30mins.
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
[jira] [Commented] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435836#comment-13435836
]
ramkrishna.s.vasudevan commented on HBASE-6587:
---
Thanks Chunhui. I am +1.
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[
https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435933#comment-13435933
]
Alex Baranau commented on HBASE-4364:
-
bq. {noformat}
return ReturnCode.NEXT_ROW; // it should be return ReturnCode.SKIP;
{noformat}
Changing to SKIP would help to get the correct results for sure. But I'd highly
doubt that in this case it will work correct internally. I.e. I believe it
would still apply filter to all columns, but because any of these applications
of filter doesn't affect the decision on whether include the whole row or not
this works. But in the essence (applying filter to all columns instead of
selected) this might work incorrect internally.
That's why I added this unit-test, which actually verifies that *filter applied
to selected columns only* (and decisions about filtering rows are based on
filters applied to selected columns only).
Filters applied to columns not in the selected column list are ignored
--
Key: HBASE-4364
URL: https://issues.apache.org/jira/browse/HBASE-4364
Project: HBase
Issue Type: Bug
Components: filters
Affects Versions: 0.90.4, 0.92.0, 0.94.0
Reporter: Todd Lipcon
Priority: Critical
Attachments:
HBASE-4364-failing-test-with-simplest-custom-filter.patch,
hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch
For a scan, if you select some set of columns using addColumns(), and then
apply a SingleColumnValueFilter that restricts the results based on some
other columns which aren't selected, then those filter conditions are ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435934#comment-13435934 ] Alex Baranau commented on HBASE-4364: - @[~grace.huang] Could you please create review on the review board? Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6575) Add SPM for HBase to Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435936#comment-13435936 ] Otis Gospodnetic commented on HBASE-6575: - @todd SPM is free, at least for now Another option is to link to a Wiki page and let people list HBase monitoring tools and services there, so HBase users have one place to look for their monitoring options. Add SPM for HBase to Ref Guide -- Key: HBASE-6575 URL: https://issues.apache.org/jira/browse/HBASE-6575 Project: HBase Issue Type: Improvement Components: documentation Reporter: Otis Gospodnetic Priority: Minor Attachments: HBASE-6575.patch Ref Guide should point users to SPM for HBase in monitoring section(s). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435805#comment-13435805
]
Zhihong Ted Yu edited comment on HBASE-6587 at 8/17/12 12:37 AM:
-
@ram
{code}
2012-08-14 20:42:54,367 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Unable to determine a plan to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
{code}
After the above log, TimeoutMonitor set allRegionServersOffline true
{code}2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available {code}
At the 2012-08-14 20:44:31, one server is onlined now, and region
277b9b6df6de2b9be1353b4fa25f4222 is sucessfully assigned.
However, at that time TimeoutMonitor, in th chore(), it would act on time out
because the if block
{code}
if (this.allRegionServersOffline !allRSsOffline) return true;
{code}
So we see the following log
{code}
2012-08-14 20:44:32,518 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OPENING for too long, reassigning
region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
The region is assigned at the time 2012-08-14 20:44:31, but is timed out by
TimeoutMonitor at the time 2012-08-14 20:44:32.
It cause the collision by two assign thread,
And the result is that the region is onlined after 30mins.
was (Author: zjushch):
@ram
{code}
2012-08-14 20:42:54,367 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Unable to determine a plan to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
{code}
After the above log, TimeoutMonitor set allRegionServersOffline true
{code}2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available {code}
At the 2012-08-14 20:44:31, one server is onlined now, and region
277b9b6df6de2b9be1353b4fa25f4222 is sucessfully assigned.
However, at that time TimeoutMonitor, in th chore(), it would act on time out
because the if block {
code}if (this.allRegionServersOffline !allRSsOffline){code} return true;
So we see the following log
{code}2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been OPENING for too long, reassigning
region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
The region is assigned at the time 2012-08-14 20:44:31, but is timed out by
TimeoutMonitor at the time 2012-08-14 20:44:32.
It cause the collision by two assign thread,
And the result is that the region is onlined after 30mins.
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and
[jira] [Commented] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435957#comment-13435957
]
Zhihong Ted Yu commented on HBASE-6587:
---
Integrated to trunk.
Thanks for the patch, Chunhui.
Thanks for the review, Ram.
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6576) HBaseAdmin.createTable should wait until the table is enabled
[
https://issues.apache.org/jira/browse/HBASE-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435965#comment-13435965
]
Zhihong Ted Yu commented on HBASE-6576:
---
Integrated to 0.92 and 0.94 as well.
Thanks for the reminder, Lars.
HBaseAdmin.createTable should wait until the table is enabled
-
Key: HBASE-6576
URL: https://issues.apache.org/jira/browse/HBASE-6576
Project: HBase
Issue Type: Bug
Components: client, test
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: HBASE-6576-92.patch, HBASE-6576-94.patch,
HBASE-6576-trunk.patch
The function:
{code}
public void createTable(final HTableDescriptor desc, byte [][] splitKeys)
{code}
in HBaseAdmin is synchronous and returns once all the regions of the table
are online, but does not wait for the table to be enabled, which is the last
step of table creation (see CreateTableHandler).
This is confusing and leads to racy code because users do not realize that
this is the case. For example, I saw the following test failure in 0.92 when
I ran:
mvn test
-Dtest=org.apache.hadoop.hbase.client.TestAdmin#testEnableDisableAddColumnDeleteColumn
{code}
Error Message
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Stacktrace
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
{code}
The issue is that code will create and table and immediately disable it in
order to do some testing, for example, to test an operation that only works
when the table is disabled. If the table has not been enabled yet, they will
get back a TableNotEnabledException.
The specific test above was fixed in HBASE-5206, but other examples exist in
the code, for example the following:
{code}
hbase org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat newtable asdf14
{code}
The code in question is:
{code}
byte[] tname = args[1].getBytes();
HTable table = util.createTable(tname, FAMILIES);
HBaseAdmin admin = new HBaseAdmin(conf);
admin.disableTable(tname);
{code}
It would be better if createTable just waited until the table was enabled, or
threw a TableNotEnabledException if it exhausted the configured number of
retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6581) Build with hadoop.profile=3.0
[ https://issues.apache.org/jira/browse/HBASE-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Charles updated HBASE-6581: Attachment: HBASE-6581-1.patch Existing hadoop-1.0 and hadoop-2.0 profiles untouched. Only changes on hadoop-3.0 profile. Install and test: mvn clean install -Dhadoop.profile=3.0 -P runSmallTests is Successful (tests also) hadoop-minicluster now defined with scope test, which is good. Package: mvn package assembly:assembly -DskipTests -Dhadoop.profile=3.0 Tests the package against hadoop 3.0 snapshot installation with simple put/get/scan via hbase shell: Success In lib folder, only needed hadoop-*: hadoop-annotations-3.0.0-SNAPSHOT.jar hadoop-auth-3.0.0-SNAPSHOT.jar hadoop-common-3.0.0-SNAPSHOT.jar hadoop-hdfs-3.0.0-SNAPSHOT.jar hadoop-mapreduce-client-common-3.0.0-SNAPSHOT.jar hadoop-mapreduce-client-core-3.0.0-SNAPSHOT.jar hadoop-yarn-api-3.0.0-SNAPSHOT.jar hadoop-yarn-common-3.0.0-SNAPSHOT.jar hadoop-yarn-server-common-3.0.0-SNAPSHOT.jar (not sure of this one) Side note: New hbase-hadoop3-compat module which imports hbase-hadoop2-compat and excludes hadoop-* jars of version 2. Sounds not logical that hbase-hadoop3-compat relies on hbase-hadoop2-compat. In fact, hbase-hadoop1-compat and hbase-hadoop2-compat could be renamed hbase-metrics1-compat and hbase-metrics2-compat as they are pure metrics related atm. Build with hadoop.profile=3.0 - Key: HBASE-6581 URL: https://issues.apache.org/jira/browse/HBASE-6581 Project: HBase Issue Type: Bug Reporter: Eric Charles Attachments: HBASE-6581-1.patch Building trunk with hadoop.profile=3.0 gives exceptions (see [1]) due to change in the hadoop maven modules naming (and also usage of 3.0-SNAPSHOT instead of 3.0.0-SNAPSHOT in hbase-common). I can provide a patch that would move most of hadoop dependencies in their respective profiles and will define the correct hadoop deps in the 3.0 profile. Please tell me if that's ok to go this way. Thx, Eric [1] $ mvn clean install -Dhadoop.profile=3.0 [INFO] Scanning for projects... [ERROR] The build could not read 3 projects - [Help 1] [ERROR] [ERROR] The project org.apache.hbase:hbase-server:0.95-SNAPSHOT (/d/hbase.svn/hbase-server/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 655, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 659, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 663, column 21 [ERROR] [ERROR] The project org.apache.hbase:hbase-common:0.95-SNAPSHOT (/d/hbase.svn/hbase-common/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 170, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 174, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 178, column 21 [ERROR] [ERROR] The project org.apache.hbase:hbase-it:0.95-SNAPSHOT (/d/hbase.svn/hbase-it/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 220, column 18 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 224, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 228, column 21 [ERROR] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5728) Methods Missing in HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435968#comment-13435968 ] Lars George commented on HBASE-5728: Sorry for being late here, but one thing that irks me is that you can set the buffer size, which already implies what the given values means, but have no means to return the buffer itself. Should we create a new JIRA that abstracts the write buffer into an interface that is returned? That way we can add it to HTableInterface as public WriteBuffer getWriteBuffer() or some such. Then the write buffer size as a long could be interpreted as whatever the actual implementation chooses? Methods Missing in HTableInterface -- Key: HBASE-5728 URL: https://issues.apache.org/jira/browse/HBASE-5728 Project: HBase Issue Type: Improvement Components: client Reporter: Bing Li Assignee: Jimmy Xiang Fix For: 0.96.0, 0.94.2 Attachments: trunk-5728.patch, trunk-5728_v2.patch, trunk-5728_v3.patch, trunk-5728_v4.patch Dear all, I found some methods existed in HTable were not in HTableInterface. setAutoFlush setWriteBufferSize ... In most cases, I manipulate HBase through HTableInterface from HTablePool. If I need to use the above methods, how to do that? I am considering writing my own table pool if no proper ways. Is it fine? Thanks so much! Best regards, Bing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435969#comment-13435969 ] Lars George commented on HBASE-6471: +1 Performance regression caused by HBASE-4054 --- Key: HBASE-6471 URL: https://issues.apache.org/jira/browse/HBASE-6471 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0 Reporter: Lars George Assignee: Jimmy Xiang Priority: Critical Fix For: 0.96.0, 0.94.2 Attachments: trunk-6471.patch, trunk-6471.patch, trunk-6471_v2.patch The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed to implement HTableInterface. Since HTable does not have an empty constructor, the patch added a call to the super() constructor, which though does trigger the ZooKeeper and META scan, causing a considerable delay. With multiple threads using the pool in parallel, the first thread is holding up all the subsequent ones, in effect it negates the whole reason we have a HTable pool. We should complete HBASE-5728, or alternatively add a protected, empty constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6595) Wrong book author name on Other HBase Resource page
Yifeng Jiang created HBASE-6595: --- Summary: Wrong book author name on Other HBase Resource page Key: HBASE-6595 URL: https://issues.apache.org/jira/browse/HBASE-6595 Project: HBase Issue Type: Bug Reporter: Yifeng Jiang Assignee: Yifeng Jiang Priority: Trivial There is a typing miss of the HBase Administration Cookbook's author name on the Other HBase Resource page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6595) Wrong book author name on Other HBase Resource page
[ https://issues.apache.org/jira/browse/HBASE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifeng Jiang updated HBASE-6595: Attachment: HBASE-6595.patch Hi, thank you for referring my book on the page. This patch fixes the typing miss of the author's name. Wrong book author name on Other HBase Resource page --- Key: HBASE-6595 URL: https://issues.apache.org/jira/browse/HBASE-6595 Project: HBase Issue Type: Bug Reporter: Yifeng Jiang Assignee: Yifeng Jiang Priority: Trivial Attachments: HBASE-6595.patch There is a typing miss of the HBase Administration Cookbook's author name on the Other HBase Resource page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435984#comment-13435984 ] Jie Huang commented on HBASE-4364: -- @Alex, totally understand your point. Yes. What you have stated in the unit-test is a problem. Not only for the functionality, but also for the better performance as Lucas mentioned. We'd better to feed all those *selected columns* to the filter. However, to make the filter only work on those selected columns doesn't help to solve the original problem here. The problem is sth. like *select A from table where B 'b'*. If we only applies the filtering criteria on column A as what we expected in the unit-test case, we will get empty result here. So basically, the solution proposed here is # To add filterred column into the selected columns, and then do the filter. The sanityCheck() function gives the chance to combine projected and where if necessary. # To remove that newly added column before returning back to the end-user. Since that newly added column is not expected by the end-user. After making the filter only work on those required columns, using suck kind of solution, we still can extend the projected by adding those where columns. And the filter may apply on required + criteria column only, instead of going through the whole column list. How about to create a new issue about the only feeds those selected columns to the filter. The initial thought is to do some comparison work in the matcher. Any comment? Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435986#comment-13435986
]
Jonathan Hsieh commented on HBASE-6586:
---
I looped the test and after about 1600 iterations it has not failed.
(TestLocaHBaseCluster).
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435986#comment-13435986
]
Jonathan Hsieh edited comment on HBASE-6586 at 8/17/12 1:12 AM:
I looped the test on 0.92.2 repo and after about 1600 iterations it has not
failed. (TestLocaHBaseCluster).
was (Author: jmhsieh):
I looped the test and after about 1600 iterations it has not failed.
(TestLocaHBaseCluster).
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6581) Build with hadoop.profile=3.0
[ https://issues.apache.org/jira/browse/HBASE-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435987#comment-13435987 ] Eric Charles commented on HBASE-6581: - Also works with m2eclipse. Import with profile='!hadoop-1.0, hadoop-3.0' Build with hadoop.profile=3.0 - Key: HBASE-6581 URL: https://issues.apache.org/jira/browse/HBASE-6581 Project: HBase Issue Type: Bug Reporter: Eric Charles Attachments: HBASE-6581-1.patch Building trunk with hadoop.profile=3.0 gives exceptions (see [1]) due to change in the hadoop maven modules naming (and also usage of 3.0-SNAPSHOT instead of 3.0.0-SNAPSHOT in hbase-common). I can provide a patch that would move most of hadoop dependencies in their respective profiles and will define the correct hadoop deps in the 3.0 profile. Please tell me if that's ok to go this way. Thx, Eric [1] $ mvn clean install -Dhadoop.profile=3.0 [INFO] Scanning for projects... [ERROR] The build could not read 3 projects - [Help 1] [ERROR] [ERROR] The project org.apache.hbase:hbase-server:0.95-SNAPSHOT (/d/hbase.svn/hbase-server/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 655, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 659, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 663, column 21 [ERROR] [ERROR] The project org.apache.hbase:hbase-common:0.95-SNAPSHOT (/d/hbase.svn/hbase-common/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 170, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 174, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 178, column 21 [ERROR] [ERROR] The project org.apache.hbase:hbase-it:0.95-SNAPSHOT (/d/hbase.svn/hbase-it/pom.xml) has 3 errors [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-common:jar is missing. @ line 220, column 18 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-annotations:jar is missing. @ line 224, column 21 [ERROR] 'dependencies.dependency.version' for org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 228, column 21 [ERROR] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435993#comment-13435993 ] Lucas Bernardi commented on HBASE-4364: --- How about specifying filterColumns explictly on the client side? So if i want to do something equivalent to select a, b from x where c0 and d1 in hbase it would be get.addColumn(a); get.addColumn(b); get.addFilterColumn(c); get.addFilterColumn(d); So the filter would only be applied on keyvalues for columns c and d. If no filetrColumn is specified, then the filter would be applied on projected columns only. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435999#comment-13435999
]
Lars Hofhansl commented on HBASE-6587:
--
Looks like a worthy addition to 0.94 as well
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436009#comment-13436009
]
Hudson commented on HBASE-6587:
---
Integrated in HBase-TRUNK #3223 (See
[https://builds.apache.org/job/HBase-TRUNK/3223/])
HBASE-6587 Region would be assigned twice in the case of all RS offline
(Chunhui) (Revision 1373829)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6576) HBaseAdmin.createTable should wait until the table is enabled
[
https://issues.apache.org/jira/browse/HBASE-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436018#comment-13436018
]
Hudson commented on HBASE-6576:
---
Integrated in HBase-0.94 #398 (See
[https://builds.apache.org/job/HBase-0.94/398/])
HBASE-6576 HBaseAdmin.createTable should wait until the table is enabled
(Gregory) (Revision 1373832)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
HBaseAdmin.createTable should wait until the table is enabled
-
Key: HBASE-6576
URL: https://issues.apache.org/jira/browse/HBASE-6576
Project: HBase
Issue Type: Bug
Components: client, test
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: HBASE-6576-92.patch, HBASE-6576-94.patch,
HBASE-6576-trunk.patch
The function:
{code}
public void createTable(final HTableDescriptor desc, byte [][] splitKeys)
{code}
in HBaseAdmin is synchronous and returns once all the regions of the table
are online, but does not wait for the table to be enabled, which is the last
step of table creation (see CreateTableHandler).
This is confusing and leads to racy code because users do not realize that
this is the case. For example, I saw the following test failure in 0.92 when
I ran:
mvn test
-Dtest=org.apache.hadoop.hbase.client.TestAdmin#testEnableDisableAddColumnDeleteColumn
{code}
Error Message
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Stacktrace
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
{code}
The issue is that code will create and table and immediately disable it in
order to do some testing, for example, to test an operation that only works
when the table is disabled. If the table has not been enabled yet, they will
get back a TableNotEnabledException.
The specific test above was fixed in HBASE-5206, but other examples exist in
the code, for example the following:
{code}
hbase org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat newtable asdf14
{code}
The code in question is:
{code}
byte[] tname = args[1].getBytes();
HTable table = util.createTable(tname, FAMILIES);
HBaseAdmin admin = new HBaseAdmin(conf);
admin.disableTable(tname);
{code}
It would be better if createTable just waited until the table was enabled, or
threw a TableNotEnabledException if it exhausted the configured number of
retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436027#comment-13436027 ] Alex Baranau commented on HBASE-4364: - @Jie Yeah, seems you are right. The problem I'm referring to is a bit different. Now that I re-read the issue description I think that two problems could be contradictory: 1. In this issue it is stated that *filters should be applied to the columns not in the selected list* 2. In the problem I'm referring to states that *filters should not be applied to the columns in the selected list* May be what I'm pointing out with the unit-test is really not a bug, but designed to be so? Though it might be not, because it would make scanner to always fetch (physically from the storage) even those columns we are not interested in if filters are added to the scan. Which might add a lot of unnecessary work (currently - when data is in different columnfamilies, or in future for same CF if the storage format will be improved somehow that will allow avoid reading columns that are not requested from single CF). I'd say it may be better to require users to specify explicitly columns to which the filters should be applied iff they are not in the selected list. In case of SingleColumnValueFilter, we may say that user specifies the column explicitly. I.e. we should also have ability, (or do we already?) for filter to add a column to that apply filter to but not transfer to the user column list. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436032#comment-13436032 ] Zhihong Ted Yu commented on HBASE-4364: --- @Alex: Take a look at SingleColumnValueExcludeFilter Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436036#comment-13436036 ] Lucas Bernardi commented on HBASE-4364: --- Yeah, I agree with Alex, the problem I se is that now, the code it somewhere in the middle, the filter is applied to the first column, and selected columns. I'm trying to figure out how to fix that, but I'm having trouble to find whi the first column is passed to the filter. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6595) Wrong book author name on Other HBase Resource page
[ https://issues.apache.org/jira/browse/HBASE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6595. -- Resolution: Fixed Sorry about that Yifeng. My fault. Fixed now. Will show on next website push. Wrong book author name on Other HBase Resource page --- Key: HBASE-6595 URL: https://issues.apache.org/jira/browse/HBASE-6595 Project: HBase Issue Type: Bug Reporter: Yifeng Jiang Assignee: Yifeng Jiang Priority: Trivial Attachments: HBASE-6595.patch There is a typing miss of the HBase Administration Cookbook's author name on the Other HBase Resource page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436040#comment-13436040 ] Alex Baranau commented on HBASE-4364: - RE SingleColumnValueExcludeFilter - then why this issue is a bug again? I believe this is the correct behavior (based on the thoughts in my last comment). Then one need to include the column is select columns list and use SingleColumnValueExcludeFilter. At the same time (different story/effort) we may want to make this apply filter but not transfer back to user feature be available in some more generic way. May be some filter wrapper or so. Filters applied to columns not in the selected column list are ignored -- Key: HBASE-4364 URL: https://issues.apache.org/jira/browse/HBASE-4364 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Todd Lipcon Priority: Critical Attachments: HBASE-4364-failing-test-with-simplest-custom-filter.patch, hbase-4364_trunk.patch, hbase-4364_trunk-v2.patch For a scan, if you select some set of columns using addColumns(), and then apply a SingleColumnValueFilter that restricts the results based on some other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6569) Extract HStore interface from Store
[ https://issues.apache.org/jira/browse/HBASE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436056#comment-13436056 ] stack commented on HBASE-6569: -- I will make a patch that renames the Interface 'Store' and the implementation 'HStore'. HStore goes with HRegion, etc. Yes, its smurf naming but I think this better than my IStore/StoreInterface and StoreImpl Extract HStore interface from Store --- Key: HBASE-6569 URL: https://issues.apache.org/jira/browse/HBASE-6569 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0 Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: 6569-v3.patch, java_HBASE-6569-v0.patch Currently Store.java is a top-level class. However, we need to to use the same interface for snapshots (HBASE-6055), and potentially other uses-cases as well, but subclassing is 'dirty' given the amount of state that a Store currently builds up on instantiation. For snapshots, we just need to modify the Store actions slightly, making a composition with interface inheritance design very appealing (meaning we need an interface for Store). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6576) HBaseAdmin.createTable should wait until the table is enabled
[
https://issues.apache.org/jira/browse/HBASE-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436067#comment-13436067
]
Hudson commented on HBASE-6576:
---
Integrated in HBase-0.92 #491 (See
[https://builds.apache.org/job/HBase-0.92/491/])
HBASE-6576 HBaseAdmin.createTable should wait until the table is enabled
(Gregory) (Revision 1373833)
Result = SUCCESS
tedyu :
Files :
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
HBaseAdmin.createTable should wait until the table is enabled
-
Key: HBASE-6576
URL: https://issues.apache.org/jira/browse/HBASE-6576
Project: HBase
Issue Type: Bug
Components: client, test
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: HBASE-6576-92.patch, HBASE-6576-94.patch,
HBASE-6576-trunk.patch
The function:
{code}
public void createTable(final HTableDescriptor desc, byte [][] splitKeys)
{code}
in HBaseAdmin is synchronous and returns once all the regions of the table
are online, but does not wait for the table to be enabled, which is the last
step of table creation (see CreateTableHandler).
This is confusing and leads to racy code because users do not realize that
this is the case. For example, I saw the following test failure in 0.92 when
I ran:
mvn test
-Dtest=org.apache.hadoop.hbase.client.TestAdmin#testEnableDisableAddColumnDeleteColumn
{code}
Error Message
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
Stacktrace
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: testMasterAdmin
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336)
{code}
The issue is that code will create and table and immediately disable it in
order to do some testing, for example, to test an operation that only works
when the table is disabled. If the table has not been enabled yet, they will
get back a TableNotEnabledException.
The specific test above was fixed in HBASE-5206, but other examples exist in
the code, for example the following:
{code}
hbase org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat newtable asdf14
{code}
The code in question is:
{code}
byte[] tname = args[1].getBytes();
HTable table = util.createTable(tname, FAMILIES);
HBaseAdmin admin = new HBaseAdmin(conf);
admin.disableTable(tname);
{code}
It would be better if createTable just waited until the table was enabled, or
threw a TableNotEnabledException if it exhausted the configured number of
retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436071#comment-13436071 ] Andrew Purtell commented on HBASE-6594: --- bq. Can the task list become long as well? [~lhofhansl] Yes, it's not bounded. However, in practice the task list is short unless the regionserver is sick. My thinking here is latency stats and top of the task list are most useful to ops at a glance with frequent refresh. The region list gets in the way in that situation. Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436071#comment-13436071 ] Andrew Purtell edited comment on HBASE-6594 at 8/17/12 3:24 AM: bq. Can the task list become long as well? [~lhofhansl] Yes, it's bounded but to IIRC 1000 entries. However, in practice the task list is short unless the regionserver is sick. My thinking here is latency stats and top of the task list are most useful to ops at a glance with frequent refresh. The region list gets in the way in that situation. was (Author: apurtell): bq. Can the task list become long as well? [~lhofhansl] Yes, it's not bounded. However, in practice the task list is short unless the regionserver is sick. My thinking here is latency stats and top of the task list are most useful to ops at a glance with frequent refresh. The region list gets in the way in that situation. Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6487) assign region doesn't check if the region is already assigned
[ https://issues.apache.org/jira/browse/HBASE-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436072#comment-13436072 ] Jimmy Xiang commented on HBASE-6487: @Stack, are you ok for the patch? I think it is good even if OFFLINE state is not so trustful. It's better not to double assign. assign region doesn't check if the region is already assigned - Key: HBASE-6487 URL: https://issues.apache.org/jira/browse/HBASE-6487 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0 Attachments: trunk-6487.patch, trunk-6487_v2.patch Tried to assign a region already assigned somewhere from hbase shell, the region is assigned to a different place but the previous assignment is not closed. So it causes double assignments. In such a case, it's better to issue a warning instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436073#comment-13436073 ] Andrew Purtell commented on HBASE-6245: --- Thanks Jie and Elliott, I'll commit patch v2. Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Priority: Minor Attachments: hbase-6245.patch, hbase-6245-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436078#comment-13436078 ] Andrew Purtell commented on HBASE-6580: --- bq. I can even see replacing the current HTablePool with this. I can see this. I believe we have a lot of users of the current HTablePool, so should be API compatible including constructors. New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor - Key: HBASE-6580 URL: https://issues.apache.org/jira/browse/HBASE-6580 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Fix For: 0.96.0, 0.94.2 Here I propose a very simple TablePool. It could be called LightHTablePool (or something - if you have a better name). Internally it would maintain an HConnection and an Executor service and each invocation of getTable(...) would create a new HTable and close() would just close it. In testing I find this more light weight than HTablePool and easier to monitor in terms of resources used. It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436081#comment-13436081
]
Andrew Purtell commented on HBASE-6586:
---
[~jmhsieh] Do you think it reasonable and worth it to add a HFile corruption
detect-and-quarantine flag and mode to hbck?
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6386) Audit log messages do not include column family / qualifier information consistently
[ https://issues.apache.org/jira/browse/HBASE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436086#comment-13436086 ] Marcelo Vanzin commented on HBASE-6386: --- Hi Andrew, Sorry but it was not clear to me, does the sound good refer to the current patch or to making the visibility changes? Audit log messages do not include column family / qualifier information consistently Key: HBASE-6386 URL: https://issues.apache.org/jira/browse/HBASE-6386 Project: HBase Issue Type: Improvement Components: security Affects Versions: 0.96.0 Reporter: Marcelo Vanzin Attachments: hbase-6386-v1.patch The code related to this issue is in AccessController.java:permissionGranted(). When creating audit logs, that method will do one of the following: * grant access, create audit log with table name only * deny access because of table permission, create audit log with table name only * deny access because of column family / qualifier permission, create audit log with specific family / qualifier So, in the case where more than one column family and/or qualifier are in the same request, there will be a loss of information. Even in the case where only one column family and/or qualifier is involved, information may be lost. It would be better if this behavior consistently included all the information in the request; regardless of access being granted or denied, and regardless which permission caused the denial, the column family and qualifier info should be part of the audit log message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6565: -- Component/s: coprocessors Affects Version/s: 0.96.0 0.92.2 Fix Version/s: 0.92.2 Summary: Coprocessor exec result Map is not thread safe (was: hbase coprocessor return the Mapbyte[],R object sometimes have a wrong size) Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436090#comment-13436090 ] Zhihong Ted Yu commented on HBASE-6565: --- @Andy: Fix version for 0.92 should be 0.92.3, I think. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436095#comment-13436095 ] Gary Helmling commented on HBASE-6565: -- I'm betting this fixes TestCoprocessorEndpoint#testAggregation, which was previously flaky. Should have seen this much sooner. I'm running TestCoprocessorEndpoint on trunk with the patch applied in a loop of 100 times to confirm. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6060: -- Status: Open (was: Patch Available) Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6060: -- Attachment: HBASE-6060_latest.patch Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5728) Methods Missing in HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436102#comment-13436102 ] Jimmy Xiang commented on HBASE-5728: Autoflush or not implies there is a buffer too. However, I think the buffer itself should be kept hidden from the interface. Otherwise, client app could mess around with it, which should be prevented, right? Methods Missing in HTableInterface -- Key: HBASE-5728 URL: https://issues.apache.org/jira/browse/HBASE-5728 Project: HBase Issue Type: Improvement Components: client Reporter: Bing Li Assignee: Jimmy Xiang Fix For: 0.96.0, 0.94.2 Attachments: trunk-5728.patch, trunk-5728_v2.patch, trunk-5728_v3.patch, trunk-5728_v4.patch Dear all, I found some methods existed in HTable were not in HTableInterface. setAutoFlush setWriteBufferSize ... In most cases, I manipulate HBase through HTableInterface from HTablePool. If I need to use the above methods, how to do that? I am considering writing my own table pool if no proper ways. Is it fine? Thanks so much! Best regards, Bing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6060: -- Status: Patch Available (was: Open) Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436101#comment-13436101 ] ramkrishna.s.vasudevan commented on HBASE-6060: --- Sorry for not updating the testcase TestOpenRegionhandler in two testcases. I had done it only one testcase. Sorry about that. Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436104#comment-13436104 ] Jonathan Leavitt commented on HBASE-6524: - Apologies, I realized that the generated protubufs should not contribute additional findbugs warnings because they should be in findbugx-exclude.xml. I believe I remember hsieh mentioning that changing the pbufs could affect findbug warnings, but maybe I misunderstood. I did however look through all the files I changed at the findbugs warnings for those files and I was unable to find anything. I will keep looking. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Attachments: htracehooks1.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6471: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks all. It is integrated into trunk and 0.94. Performance regression caused by HBASE-4054 --- Key: HBASE-6471 URL: https://issues.apache.org/jira/browse/HBASE-6471 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0 Reporter: Lars George Assignee: Jimmy Xiang Priority: Critical Fix For: 0.96.0, 0.94.2 Attachments: trunk-6471.patch, trunk-6471.patch, trunk-6471_v2.patch The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed to implement HTableInterface. Since HTable does not have an empty constructor, the patch added a call to the super() constructor, which though does trigger the ZooKeeper and META scan, causing a considerable delay. With multiple threads using the pool in parallel, the first thread is holding up all the subsequent ones, in effect it negates the whole reason we have a HTable pool. We should complete HBASE-5728, or alternatively add a protected, empty constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436106#comment-13436106
]
Jonathan Hsieh commented on HBASE-6586:
---
Looks like if we go with the hbck approach instead of the autofix approach,
we'll should to add yet another state for a region which I'll call something
like QUARANTINED. Currently, corrupted HFiles cause a region to go into
RS_ZK_REGION_FAILED_OPEN, which eventually gets transitioned to
M_ZK_REGION_OFFLINE which then transitions to RS_ZK_REGION_OPENING triggering
another attempt to open the region (which fails and ...). Ideally when we have
a none-recoverable failure like corrupted hfiles, we'd transition to
QUARANTINED instead of FAILED_OPEN and stay there until an admin fixes the
problem.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
stack created HBASE-6596: Summary: Revert HBASE-5022; it undoes HBC.create Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436113#comment-13436113
]
Jonathan Hsieh commented on HBASE-6586:
---
[~apurtell] I do, I built it yesterday and am polishing up the test currently,
and looking for more corner cases. It is a separate phase (it ignores
metadata) and is kind of analogous to checking every block on a file system.
The patch is about 2x the size of the previous just fix version.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436116#comment-13436116 ] Andrew Purtell commented on HBASE-6565: --- @Gary, yes doing that here too precommit. Please put up your results too. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Labels: coprocessors, patch Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6565: -- Fix Version/s: (was: 0.92.2) 0.92.3 @Ted updated fix version for 0.92, thanks! Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Labels: coprocessors, patch Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436118#comment-13436118
]
Andrew Purtell commented on HBASE-6586:
---
bq. I do, I built it yesterday and am polishing up the test currently, and
looking for more corner cases. It is a separate phase (it ignores metadata) and
is kind of analogous to checking every block on a file system
Sounds great! Thanks Jon.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436120#comment-13436120
]
Andrew Purtell commented on HBASE-6586:
---
bq. Ideally when we have a none-recoverable failure like corrupted hfiles, we'd
transition to QUARANTINED instead of FAILED_OPEN and stay there until an admin
fixes the problem.
+1 on this too.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu reassigned HBASE-6565: - Assignee: Yuan Kang Nice catch, Yuan. Assigning issue to you. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436126#comment-13436126 ] Lars Hofhansl commented on HBASE-6594: -- Fair enough. +1 on patch. Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6587) Region would be assigned twice in the case of all RS offline
[
https://issues.apache.org/jira/browse/HBASE-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6587:
-
Fix Version/s: 0.94.2
Region would be assigned twice in the case of all RS offline
Key: HBASE-6587
URL: https://issues.apache.org/jira/browse/HBASE-6587
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.2
Attachments: 6587.patch, HBASE-6587.patch
In the TimeoutMonitor, we would act on time out for the regions if
(this.allRegionServersOffline !noRSAvailable)
The code is as the following:
{code}
if (regionState.getStamp() + timeout = now ||
(this.allRegionServersOffline !noRSAvailable)) {
//decide on action upon timeout or, if some RSs just came back
online, we can start the
// the assignment
actOnTimeOut(regionState);
}
{code}
But we found it exists a bug that it would act on time out for the region
which was assigned just now , and cause assigning the region twice.
Master log for the region 277b9b6df6de2b9be1353b4fa25f4222:
{code}
2012-08-14 20:42:54,367 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Unable to determine a plan
to assign .META.,,1.1028785192 state=OFFLINE, ts=1
344948174367, server=null
2012-08-14 20:44:31,640 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
was found (or we are ignoring an existing plan) for writete
st,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa25f4222. so
generated a random one;
hri=writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be13
53b4fa25f4222., src=, dest=dw92.kgb.sqa.cm4,60020,1344948267642; 1 (online=1,
available=1) available servers
2012-08-14 20:44:31,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x438f53bbf9b0acd Creating (or updating) unassigned node for
277b9b6df6de2b9be13
53b4fa25f4222 with OFFLINE state
2012-08-14 20:44:31,643 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df6de2b9be1353b4fa
25f4222. to dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=dw92.kgb.sqa.cm4,60020,1344948267642,
region=277b9b6df6de2b9be1353b4fa25f4222
// 异常的超时
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: writetest,VHXYHJN0BL48HMR4DI1L,1344925649429.277b9b6df
6de2b9be1353b4fa25f4222. state=OPENING, ts=1344948272279,
server=dw92.kgb.sqa.cm4,60020,1344948267642
2012-08-14 20:44:32,518 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for
too long, reassigning region=writetest,VHXYHJN0BL48HMR4DI1L,
1344925649429.277b9b6df6de2b9be1353b4fa25f4222.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6596: - Attachment: 6596_094.txt hbase-5022 applied in reverse Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Attachments: 6596_094.txt, 6596_trunk.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6596: - Attachment: 6596_trunk.txt -R of hbase-5022 for trunk Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Attachments: 6596_094.txt, 6596_trunk.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6596. -- Resolution: Fixed Fix Version/s: 0.94.2 Assignee: stack Applied to 0.94 and trunk. Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436140#comment-13436140 ] stack commented on HBASE-5022: -- I reverted this patch. See HBASE-6596. N, when you get a chance, can you take a look at getting same perf otherwise. Also, this issue is w/o explanation of how this change improved performance. Its makes it hard justifying why this patch should be retained at all. Thanks boss. Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6593) TestAdmin times out sometimes
[ https://issues.apache.org/jira/browse/HBASE-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436147#comment-13436147 ] David S. Wang commented on HBASE-6593: -- +1 TestAdmin times out sometimes - Key: HBASE-6593 URL: https://issues.apache.org/jira/browse/HBASE-6593 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0, 0.94.2 Attachments: trunk-6593.patch In TestAdmin#splitTest, individual put is used to prepare the test data. We can group them together so as to avoid possible timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6565: - Fix Version/s: (was: 0.92.3) 0.92.2 Marking as 0.92.2 since this patch has been applied to 0.92 branch and 0.92.2 is not yet released. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436150#comment-13436150 ] Andrew Purtell commented on HBASE-6565: --- @stack It hasn't been committed yet, still waiting for 100 consecutive runs to pass, but I expect it to go in, will commit on 0.92 branch too. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6593) TestAdmin times out sometimes
[ https://issues.apache.org/jira/browse/HBASE-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436151#comment-13436151 ] Jimmy Xiang commented on HBASE-6593: Thanks David for review. I will commit it tomorrow if no objection. TestAdmin times out sometimes - Key: HBASE-6593 URL: https://issues.apache.org/jira/browse/HBASE-6593 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0, 0.94.2 Attachments: trunk-6593.patch In TestAdmin#splitTest, individual put is used to prepare the test data. We can group them together so as to avoid possible timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6561) Gets/Puts with many columns send the RegionServer into an endless loop
[ https://issues.apache.org/jira/browse/HBASE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6561: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.94 and 0.96. Gets/Puts with many columns send the RegionServer into an endless loop Key: HBASE-6561 URL: https://issues.apache.org/jira/browse/HBASE-6561 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 6561-0.94.txt, 6561-0.96.txt, 6561-0.96-v2.txt, 6561-0.96-v3.txt, 6561-0.96-v4.txt, 6561-0.96-v4.txt, 6561-0.96-v5.txt This came from the mailing this: We were able to replicate this behavior in a pseudo-distributed hbase (hbase-0.94.1) environment. We wrote a test program that creates a test table MyTestTable and populates it with random rows, then it creates a row with 60,000 columns and repeatedly updates it. Each column has a 18 byte qualifier and a 50 byte value. In our tests, when we ran the program, we usually never got beyond 15 updates before it would flush for a really long time. The rows that are being updated are about 4MB each (minues any hbase metadata). It doesn't seem like it's caused by GC. I turned on gc logging, and didn't see any long pauses. This is the gc log during the flush. http://pastebin.com/vJKKXDx5 This is the regionserver log with debug on during the same flush http://pastebin.com/Fh5213mg This is the test program we wrote. http://pastebin.com/aZ0k5tx2 You should be able to just compile it, and run it against a running HBase cluster. $ java TestTable Carlos -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436155#comment-13436155 ] stack commented on HBASE-6565: -- @Andrew OK. Thanks. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436159#comment-13436159 ] Hadoop QA commented on HBASE-6060: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541239/HBASE-6060_latest.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2598//console This message is automatically generated. Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.2 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see:
[jira] [Commented] (HBASE-4054) Usability improvement to HTablePool
[ https://issues.apache.org/jira/browse/HBASE-4054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436160#comment-13436160 ] Hudson commented on HBASE-4054: --- Integrated in HBase-0.94 #399 (See [https://builds.apache.org/job/HBase-0.94/399/]) HBASE-6471 Performance regression caused by HBASE-4054 (Revision 1373916) Result = FAILURE jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java Usability improvement to HTablePool --- Key: HBASE-4054 URL: https://issues.apache.org/jira/browse/HBASE-4054 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.90.3 Reporter: Daniel Iancu Assignee: Daniel Iancu Priority: Minor Fix For: 0.92.0 Attachments: 4054.addendum, HBASE-4054_Usability_improvement_to_HTablePool.patch, HBASE-4054_Usability_improvement_to_HTablePool-trunk.patch, TestHTablePool-trunk.patch To improve the usability of the HTablePool the implementation should not rely on the user returning the connection to the pool but rather do that transparently when user closes the HTableImplementation it got. To do that a HTableImplementation proxy implementation should be returned that wraps a HTable object and holds a reference to the pool. When the client close the proxy it will actually automatically return the wrapped HTable back in pool to be reused. In this case the method HTablePool.putTable don't need to be public -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6471) Performance regression caused by HBASE-4054
[ https://issues.apache.org/jira/browse/HBASE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436161#comment-13436161 ] Hudson commented on HBASE-6471: --- Integrated in HBase-0.94 #399 (See [https://builds.apache.org/job/HBase-0.94/399/]) HBASE-6471 Performance regression caused by HBASE-4054 (Revision 1373916) Result = FAILURE jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java Performance regression caused by HBASE-4054 --- Key: HBASE-6471 URL: https://issues.apache.org/jira/browse/HBASE-6471 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0 Reporter: Lars George Assignee: Jimmy Xiang Priority: Critical Fix For: 0.96.0, 0.94.2 Attachments: trunk-6471.patch, trunk-6471.patch, trunk-6471_v2.patch The patch in HBASE-4054 switches the PooledHTable to extend HTable as opposed to implement HTableInterface. Since HTable does not have an empty constructor, the patch added a call to the super() constructor, which though does trigger the ZooKeeper and META scan, causing a considerable delay. With multiple threads using the pool in parallel, the first thread is holding up all the subsequent ones, in effect it negates the whole reason we have a HTable pool. We should complete HBASE-5728, or alternatively add a protected, empty constructor the HTable. I am +1 for the former. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436165#comment-13436165 ] Andrew Purtell commented on HBASE-6594: --- Ok will commit soon unless objection. Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436169#comment-13436169 ] Gary Helmling commented on HBASE-6565: -- Ran TestCoprocessorEndpoint on trunk 100 times with the patch applied and testAggregation() re-enabled. No failures. Previously testAggregation() would fail reliably withing ~50 runs. Looks like this patch fixes the issue. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436172#comment-13436172 ] Gary Helmling commented on HBASE-6565: -- So, just to be clear, big +1 from me. Thanks Yuan! Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6593) TestAdmin times out sometimes
[ https://issues.apache.org/jira/browse/HBASE-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436181#comment-13436181 ] Jonathan Hsieh commented on HBASE-6593: --- lgtm. +1. TestAdmin times out sometimes - Key: HBASE-6593 URL: https://issues.apache.org/jira/browse/HBASE-6593 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0, 0.94.2 Attachments: trunk-6593.patch In TestAdmin#splitTest, individual put is used to prepare the test data. We can group them together so as to avoid possible timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6597) Block Encoding Size Estimation
Brian Nixon created HBASE-6597: -- Summary: Block Encoding Size Estimation Key: HBASE-6597 URL: https://issues.apache.org/jira/browse/HBASE-6597 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.89-fb Reporter: Brian Nixon Priority: Minor Blocks boundaries as created by current writers are determined by the size of the unencoded data. However, blocks in memory are kept encoded. By using an estimate for the encoded size of the block, we can get greater consistency in size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5796) Fix our abuse of IOE: see http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html
[ https://issues.apache.org/jira/browse/HBASE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436208#comment-13436208 ] Jonathan Hsieh commented on HBASE-5796: --- HBaseException.java and DeserializationException.java snuck in with HBASE-5336 I'm working on another patch (HBASE-6586) and adding am considering adding an HBaseIOException that is a subclass of IOException. This would allow for compatibility with older versions for the time being and would allow the code to be more readable if used properly. Fix our abuse of IOE: see http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html --- Key: HBASE-5796 URL: https://issues.apache.org/jira/browse/HBASE-5796 Project: HBase Issue Type: Task Reporter: stack Lets make more context particular exceptions rather than throw IOEs everywhere. See Benoît's rant: http://blog.tsunanet.net/2012/04/apache-hadoop-abuse-ioexception.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6561) Gets/Puts with many columns send the RegionServer into an endless loop
[ https://issues.apache.org/jira/browse/HBASE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436215#comment-13436215 ] Hudson commented on HBASE-6561: --- Integrated in HBase-TRUNK #3226 (See [https://builds.apache.org/job/HBase-TRUNK/3226/]) HBASE-6561 Gets/Puts with many columns send the RegionServer into an 'endless' loop (Revision 1373943) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java Gets/Puts with many columns send the RegionServer into an endless loop Key: HBASE-6561 URL: https://issues.apache.org/jira/browse/HBASE-6561 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 6561-0.94.txt, 6561-0.96.txt, 6561-0.96-v2.txt, 6561-0.96-v3.txt, 6561-0.96-v4.txt, 6561-0.96-v4.txt, 6561-0.96-v5.txt This came from the mailing this: We were able to replicate this behavior in a pseudo-distributed hbase (hbase-0.94.1) environment. We wrote a test program that creates a test table MyTestTable and populates it with random rows, then it creates a row with 60,000 columns and repeatedly updates it. Each column has a 18 byte qualifier and a 50 byte value. In our tests, when we ran the program, we usually never got beyond 15 updates before it would flush for a really long time. The rows that are being updated are about 4MB each (minues any hbase metadata). It doesn't seem like it's caused by GC. I turned on gc logging, and didn't see any long pauses. This is the gc log during the flush. http://pastebin.com/vJKKXDx5 This is the regionserver log with debug on during the same flush http://pastebin.com/Fh5213mg This is the test program we wrote. http://pastebin.com/aZ0k5tx2 You should be able to just compile it, and run it against a running HBase cluster. $ java TestTable Carlos -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436216#comment-13436216 ] Hudson commented on HBASE-5022: --- Integrated in HBase-TRUNK #3226 (See [https://builds.apache.org/job/HBase-TRUNK/3226/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create (Revision 1373936) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436217#comment-13436217 ] Hudson commented on HBASE-6596: --- Integrated in HBase-TRUNK #3226 (See [https://builds.apache.org/job/HBase-TRUNK/3226/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create (Revision 1373936) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6565: -- Tags: coprocessor (was: coprocessor HTable) Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, 0.94, and 0.92 branches. TestCoprocessorEndpoint passes 100 times consecutively locally. Thanks for the patch Yuan. Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436222#comment-13436222 ] Hudson commented on HBASE-5022: --- Integrated in HBase-0.94 #400 (See [https://builds.apache.org/job/HBase-0.94/400/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create (Revision 1373935) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436223#comment-13436223 ] Hudson commented on HBASE-6596: --- Integrated in HBase-0.94 #400 (See [https://builds.apache.org/job/HBase-0.94/400/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create (Revision 1373935) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6561) Gets/Puts with many columns send the RegionServer into an endless loop
[ https://issues.apache.org/jira/browse/HBASE-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436221#comment-13436221 ] Hudson commented on HBASE-6561: --- Integrated in HBase-0.94 #400 (See [https://builds.apache.org/job/HBase-0.94/400/]) HBASE-6561 Gets/Puts with many columns send the RegionServer into an 'endless' loop (Revision 1373951) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java Gets/Puts with many columns send the RegionServer into an endless loop Key: HBASE-6561 URL: https://issues.apache.org/jira/browse/HBASE-6561 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 6561-0.94.txt, 6561-0.96.txt, 6561-0.96-v2.txt, 6561-0.96-v3.txt, 6561-0.96-v4.txt, 6561-0.96-v4.txt, 6561-0.96-v5.txt This came from the mailing this: We were able to replicate this behavior in a pseudo-distributed hbase (hbase-0.94.1) environment. We wrote a test program that creates a test table MyTestTable and populates it with random rows, then it creates a row with 60,000 columns and repeatedly updates it. Each column has a 18 byte qualifier and a 50 byte value. In our tests, when we ran the program, we usually never got beyond 15 updates before it would flush for a really long time. The rows that are being updated are about 4MB each (minues any hbase metadata). It doesn't seem like it's caused by GC. I turned on gc logging, and didn't see any long pauses. This is the gc log during the flush. http://pastebin.com/vJKKXDx5 This is the regionserver log with debug on during the same flush http://pastebin.com/Fh5213mg This is the test program we wrote. http://pastebin.com/aZ0k5tx2 You should be able to just compile it, and run it against a running HBase cluster. $ java TestTable Carlos -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6594. --- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Committed to trunk. Thanks for the review Lars. Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.96.0 Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-6245: - Assignee: Jie Huang Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Jie Huang Priority: Minor Attachments: hbase-6245.patch, hbase-6245-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[
https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436232#comment-13436232
]
Zhihong Ted Yu commented on HBASE-6596:
---
Looks like TestClassLoading failure in trunk is reproducible.
In test log, I saw:
{code}
2012-08-16 11:59:55,134 WARN
[RS_OPEN_REGION-lm-sjn-00713032.corp.ebay.com,61729,1345143569962-2]
regionserver.RegionCoprocessorHost(201):
java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.iterator(CompoundConfiguration.java:408)
at
org.apache.hadoop.hbase.HBaseConfiguration.merge(HBaseConfiguration.java:135)
at
org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:124)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:183)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.lt;initgt;(RegionCoprocessorHost.java:152)
at org.apache.hadoop.hbase.regionserver.HRegion.lt;initgt;(HRegion.java:461)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3715)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3896)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:334)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:110)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:195)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
{code}
Revert HBASE-5022; it undoes HBC.create
---
Key: HBASE-6596
URL: https://issues.apache.org/jira/browse/HBASE-6596
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.94.2
Attachments: 6596_094.txt, 6596_trunk.txt
See this thread:
http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6245) Upon page refresh new UI should return to the previously selected tab
[ https://issues.apache.org/jira/browse/HBASE-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-6245. --- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Committed to trunk. Thanks for the patch Jie. Thanks for the review Elliott. Upon page refresh new UI should return to the previously selected tab - Key: HBASE-6245 URL: https://issues.apache.org/jira/browse/HBASE-6245 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Jie Huang Priority: Minor Fix For: 0.96.0 Attachments: hbase-6245.patch, hbase-6245-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6594) Move tasks section above regions section in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436257#comment-13436257 ] Hudson commented on HBASE-6594: --- Integrated in HBase-TRUNK #3227 (See [https://builds.apache.org/job/HBase-TRUNK/3227/]) HBASE-6594. Move tasks section above regions section in RegionServer UI (Revision 1373980) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon Move tasks section above regions section in RegionServer UI --- Key: HBASE-6594 URL: https://issues.apache.org/jira/browse/HBASE-6594 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.96.0 Attachments: HBASE-6594.patch With the new RegionServer UI, at the top of the page is the server metrics tab, then the region list, then the task list, the software attributes. The region list could be lengthy, so scrolling down to find the task list can take some time. Every refresh of the page resets the view to [0,0]. Therefore at a glance information should come first, above the fold, statistics at the top followed by the tasks section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436258#comment-13436258 ] Hudson commented on HBASE-6565: --- Integrated in HBase-TRUNK #3227 (See [https://builds.apache.org/job/HBase-TRUNK/3227/]) HBASE-6565. Coprocessor exec result Map is not thread safe (Yuan Kang) (Revision 1373972) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6596: - Attachment: addendum.txt Thanks Ted. Here is addendum I applied to trunk and 0.94. It changes RegionCoprocessorHost to do new Configuration(conf) instead of HBaseConfiguration.create(conf) instead of changing how HBC.create(conf) works. Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt, addendum.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436268#comment-13436268 ] Hudson commented on HBASE-6565: --- Integrated in HBase-0.94 #401 (See [https://builds.apache.org/job/HBase-0.94/401/]) HBASE-6565. Coprocessor exec result Map is not thread safe (Yuan Kang) (Revision 1373975) Result = SUCCESS apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6593) TestAdmin times out sometimes
[ https://issues.apache.org/jira/browse/HBASE-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6593: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Jon and Dave for the review. Integrated into trunk and 0.94. TestAdmin times out sometimes - Key: HBASE-6593 URL: https://issues.apache.org/jira/browse/HBASE-6593 Project: HBase Issue Type: Test Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0, 0.94.2 Attachments: trunk-6593.patch In TestAdmin#splitTest, individual put is used to prepare the test data. We can group them together so as to avoid possible timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6586) Quarantine Corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6586:
--
Attachment: hbase-6586.patch
Version for trunk. Tested locally, seems to pass. Will post on review board,
and after it make it through there I plan to backport to 0.94/0.92.
Quarantine Corrupted HFiles
---
Key: HBASE-6586
URL: https://issues.apache.org/jira/browse/HBASE-6586
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6586.patch
We've encountered a few upgrades from 0.90 hbases + 20.2/1.x hdfs to 0.92
hbases + hdfs 2.x that get stuck. I haven't been able to duplicate the
problem in my dev environment but we suspect this may be related to
HDFS-3731. On the HBase side, it seems reasonable to quarantine what are
most likely truncated hfiles, so that can could later be recovered.
Here's an example of the exception we've encountered:
{code}
2012-07-18 05:55:01,152 ERROR handler.OpenRegionHandler
(OpenRegionHandler.java:openRegion(346)) - Failed open of
region=user_mappings,080112102AA76EF98197605D341B9E6C5824D2BC|1001,1317824890618.eaed0e7abc6d27d28ff0e5a9b49c4c
0d.
java.io.IOException: java.lang.IllegalArgumentException: Invalid HFile
version: 842220600 (expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:306)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:371)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:387)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1026)
at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:485)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:566)
at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:286)
at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:223)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2534)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:454)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3282)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3230)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
842220600 (expected to be between 1 and 2)
at org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:515)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:303)
... 17 more
{code}
Specifically -- the FixedFileTrailer are incorrect, and seemingly missing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436316#comment-13436316 ] nkeywal commented on HBASE-5022: It was mainly mentioned in HBASE-4993 and HBASE-5027. Basically, the comment in create was: * Creates a clone of passed configuration. * @param that Configuration to clone. * @return a Configuration created with the hbase-*.xml files plus the given configuration. This comment was saying two different things (clone vs. load+merge), and I've chosen the most efficient one (clone). As actually it was mostly used as a clone function (while it was not a clone function) so it worked (mostly :-) ). So imho it makes sense to revert this patch to keep the load+mergefunction, but then we should remove the first comment:* Creates a clone of passed configuration.. Ideally, we should review how it's used in HBase today to understand when they want a clone vs. when they want a load+merge... But well... Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6565) Coprocessor exec result Map is not thread safe
[ https://issues.apache.org/jira/browse/HBASE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436321#comment-13436321 ] Hudson commented on HBASE-6565: --- Integrated in HBase-0.92 #493 (See [https://builds.apache.org/job/HBase-0.92/493/]) HBASE-6565. Coprocessor exec result Map is not thread safe (Yuan Kang) (Revision 1373976) Result = SUCCESS apurtell : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java Coprocessor exec result Map is not thread safe -- Key: HBASE-6565 URL: https://issues.apache.org/jira/browse/HBASE-6565 Project: HBase Issue Type: Bug Components: client, coprocessors Affects Versions: 0.92.2, 0.94.0, 0.96.0 Environment: hadoop1.0.2,hbase0.94,jdk1.6 Reporter: Yuan Kang Assignee: Yuan Kang Labels: coprocessors, patch Fix For: 0.92.2, 0.96.0, 0.94.2 Attachments: Coprocessor-result-thread unsafe-bug-fix.patch Original Estimate: 168h Remaining Estimate: 168h I develop a coprocessor program ,but found some different results in repeated tests.for example,normally,the result's size is 10.but sometimes it appears 9. I read the HTable.java code,found a TreeMap(thread-unsafe) be used in multithreading environment.It cause the bug happened -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436331#comment-13436331 ] Hudson commented on HBASE-6596: --- Integrated in HBase-0.94 #402 (See [https://builds.apache.org/job/HBase-0.94/402/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create -- ADDENDUM (Revision 1374013) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt, addendum.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436330#comment-13436330 ] Hudson commented on HBASE-5022: --- Integrated in HBase-0.94 #402 (See [https://builds.apache.org/job/HBase-0.94/402/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create -- ADDENDUM (Revision 1374013) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436335#comment-13436335 ] Jean-Daniel Cryans commented on HBASE-5022: --- bq. Ideally, we should review how it's used in HBase today to understand when they want a clone vs. when they want a load+merge... But well... As I mentioned on dev@, at SU it's being used to add hbsae-site.xml but doing addHbaseResources works just as well. I would say that cloning + adding hbase-site.xml is also a valid use case, but the most important thing to me is that this is a *public* *API* so just looking at how HBase is using it is not enough, and you can't really ask all the users either. That's why I'm in favor of using the old behavior or deleting the method entirely so that instead of creating weird bugs for users it will just stop compiling. Obviously the former is less agressive. Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6596) Revert HBASE-5022; it undoes HBC.create
[ https://issues.apache.org/jira/browse/HBASE-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13436342#comment-13436342 ] Hudson commented on HBASE-6596: --- Integrated in HBase-TRUNK #3228 (See [https://builds.apache.org/job/HBase-TRUNK/3228/]) HBASE-6596 Revert HBASE-5022; it undoes HBC.create -- ADDENDUM (Revision 1374012) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java Revert HBASE-5022; it undoes HBC.create --- Key: HBASE-6596 URL: https://issues.apache.org/jira/browse/HBASE-6596 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.2 Attachments: 6596_094.txt, 6596_trunk.txt, addendum.txt See this thread: http://search-hadoop.com/m/WAXXV1OQ1QY/What+is+HBaseConfiguration.create%2528Configuration%2529+good+for%253Fsubj=What+is+HBaseConfiguration+create+Configuration+good+for+ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
