[jira] [Commented] (HBASE-7230) port HBASE-7109 integration tests on cluster are not getting picked up from distribution to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506305#comment-13506305 ] Hudson commented on HBASE-7230: --- Integrated in HBase-0.94 #606 (See [https://builds.apache.org/job/HBase-0.94/606/]) HBASE-7230 port HBASE-7109 integration tests on cluster are not getting picked up from distribution to 0.94 (Revision 1415054) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ClassFinder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java port HBASE-7109 integration tests on cluster are not getting picked up from distribution to 0.94 Key: HBASE-7230 URL: https://issues.apache.org/jira/browse/HBASE-7230 Project: HBase Issue Type: Bug Affects Versions: 0.94.4 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.94.4 Attachments: HBASE-7109-v6-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7109) integration tests on cluster are not getting picked up from distribution
[ https://issues.apache.org/jira/browse/HBASE-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506304#comment-13506304 ] Hudson commented on HBASE-7109: --- Integrated in HBase-0.94 #606 (See [https://builds.apache.org/job/HBase-0.94/606/]) HBASE-7230 port HBASE-7109 integration tests on cluster are not getting picked up from distribution to 0.94 (Revision 1415054) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ClassFinder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestCheckTestClasses.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/TestClassFinder.java integration tests on cluster are not getting picked up from distribution Key: HBASE-7109 URL: https://issues.apache.org/jira/browse/HBASE-7109 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-7109-fix.patch, HBASE-7109-fix-v2.patch, HBASE-7109-squashed.patch, HBASE-7109-v2-squashed.patch, HBASE-7109-v3-squashed.patch, HBASE-7109-v4-squashed.patch, HBASE-7109-v5.patch, HBASE-7109-v5.patch, HBASE-7109-v6-0.94.patch, HBASE-7109-v6.patch The method of finding test classes only works on local build (or its full copy), not if the distribution is used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506314#comment-13506314 ] Ted Yu commented on HBASE-7213: --- I wonder what kind of data structure should be in place for the FSHLog instances (two for the region server hosting .META.) so that multiple WAL implementation would be more intuitive. Previously, close / deletion of HLog operates on the single instance. Now things start to get more interesting. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6973) Trim trailing whitespace from configuration values
[
https://issues.apache.org/jira/browse/HBASE-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506357#comment-13506357
]
Ted Yu commented on HBASE-6973:
---
If the fix isn't in hadoop 1.0 / 1.1 (default hadoop relase for HBase), we
should fix this in HBase.
Trim trailing whitespace from configuration values
--
Key: HBASE-6973
URL: https://issues.apache.org/jira/browse/HBASE-6973
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1
Reporter: James Kinley
Priority: Minor
If the following properties have a whitespace at the end of the value it
causes problems with Kerberos authentication...
{code}
property
namehbase.regionserver.keytab.file/name
value/etc/hbase/conf/hbase.krb5 /value
/property
property
namehbase.master.keytab.file/name
value/etc/hbase/conf/hbase.krb5 /value
/property
{code}
I'm guessing this can affect other properties as well, so can the values be
trimmed when read?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2039) G1 GC issues
[ https://issues.apache.org/jira/browse/HBASE-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506374#comment-13506374 ] liang xie commented on HBASE-2039: -- en, it's not recommented to enable G1 before jdk7u4 we need to reevaluate this issue with jdk7u4+ again,IMHO G1 GC issues Key: HBASE-2039 URL: https://issues.apache.org/jira/browse/HBASE-2039 Project: HBase Issue Type: Sub-task Reporter: stack Lets keep an issue where we report on g1 issues. Lets keep list of crashes we see. I filed an issue up on bug parade. Lets see if it becomes actual bug. Below I note version of vm and the type of crash (Internal Error (nmethod.cpp:1981), pid=32319. Its a 'fatal error'). It happens for me after 5-10 minutes when a loading test. Same thing each time. G1 in 1.6 seems plain broke; crashes on use of stuff in concurrent utils package. Date Created: Thu Dec 10 13:33:04 MST 2009 .. Synopsis:Running G1 GC, crashes with Internal Error (nmethod.cpp:1981), pid=32319... Description: FULL PRODUCT VERSION : java version 1.7.0-ea Java(TM) SE Runtime Environment (build 1.7.0-ea-b77) Java HotSpot(TM) 64-Bit Server VM (build 17.0-b05, mixed mode) FULL OS VERSION : Fedora Core release 6 (Zod) EXTRA RELEVANT SYSTEM CONFIGURATION : Here are the JVM args: -XX:+HeapDumpOnOutOfMemoryError -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1648) heap usage and limit reporting doesn't work as expected when using G1 GC
[
https://issues.apache.org/jira/browse/HBASE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506375#comment-13506375
]
liang xie commented on HBASE-1648:
--
emmm, it's not a hbase related bug, some jdk tools don't work well with G1 on
jdk6
heap usage and limit reporting doesn't work as expected when using G1 GC
Key: HBASE-1648
URL: https://issues.apache.org/jira/browse/HBASE-1648
Project: HBase
Issue Type: Bug
Environment: Linux Centos x86_64 5.3, Sun JDK 1.6.0_14 (HotSpot
64-Bit Server VM build 14.0-b16)
Reporter: Andrew Purtell
Priority: Minor
Getting some early experience with the G1 GC.
Running with -Xmx1000m and HBASE_OPTS set to -XX:+UnlockExperimentalVMOpts
-XX:+UseG1GC. Regionserver heap use reports are not of much use:
{noformat}
hbase(main):001:0 status 'simple'
3 live servers
test3:60020 1247389283042
requests=0, regions=1, usedHeap=0, maxHeap=41
test2:60020 1247389219994
requests=0, regions=1, usedHeap=0, maxHeap=41
test4:60020 1247389324563
requests=0, regions=2, usedHeap=0, maxHeap=41
0 dead servers
{noformat}
top is about right:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
23988 hadoop24 0 1492m 98m 10m S 0.0 2.5 0:02.65 java
{noformat}
Incidentally, don't try -XX:+UseG1GC and -XX:+DoEscapeAnalysis together or
the JVM will rapidly segfault.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7220) Creating a table with 3000 regions on 2 nodes fails after 1 hour
[
https://issues.apache.org/jira/browse/HBASE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506424#comment-13506424
]
Hudson commented on HBASE-7220:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #279 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/279/])
HBASE-7220 Creating a table with 3000 regions on 2 nodes fails after 1 hour
(Revision 1415016)
Result = FAILURE
eclark :
Files :
*
/hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/impl/JmxCacheBuster.java
*
/hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/impl/JmxCacheBuster.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Creating a table with 3000 regions on 2 nodes fails after 1 hour
Key: HBASE-7220
URL: https://issues.apache.org/jira/browse/HBASE-7220
Project: HBase
Issue Type: Bug
Components: metrics, Performance, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: Elliott Clark
Attachments: HBASE-7220-0.patch, HBASE-7220-1.patch,
HBASE-7220-2.patch
I'm trying to create a table with 3000 regions on two regions servers, from
the shell.
It's ok on trunk a standalone config.
It's ok on 0.94
It's not ok on trunk: it fails after around 1 hour.
If I remove all the code related to metrics in HRegion, the 3000 regions are
created in 3 minutes (twice faster than the 0.94).
On trunk, the region server spends its time in waitForWork, while the
master is in the tcp connection related code. It's a 1Gb network.
I haven't looked at the metric code itself.
Patch used to remove the metrics from HRegion:
{noformat}
index c70e9ab..6677e65 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -364,7 +364,7 @@ public class HRegion implements HeapSize { // , Writable{
private HTableDescriptor htableDescriptor = null;
private RegionSplitPolicy splitPolicy;
- private final MetricsRegion metricsRegion;
+ private final MetricsRegion metricsRegion = null;
/**
* Should only be used for testing purposes
@@ -388,7 +388,7 @@ public class HRegion implements HeapSize { // , Writable{
this.coprocessorHost = null;
this.scannerReadPoints = new ConcurrentHashMapRegionScanner, Long();
-this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
+//this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
}
/**
@@ -451,7 +451,7 @@ public class HRegion implements HeapSize { // , Writable{
this.regiondir = getRegionDir(this.tableDir, encodedNameStr);
this.scannerReadPoints = new ConcurrentHashMapRegionScanner, Long();
-this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
+//this.metricsRegion = new MetricsRegion(new
MetricsRegionWrapperImpl(this));
/*
* timestamp.slop provides a server-side constraint on the timestamp.
This
@@ -1024,7 +1024,7 @@ public class HRegion implements HeapSize { // ,
Writable{
status.setStatus(Running coprocessor post-close hooks);
this.coprocessorHost.postClose(abort);
}
- this.metricsRegion.close();
+ //this.metricsRegion.close();
status.markComplete(Closed);
LOG.info(Closed + this);
return result;
@@ -2331,11 +2331,11 @@ public class HRegion implements HeapSize { // ,
Writable{
if (noOfPuts 0) {
// There were some Puts in the batch.
double noOfMutations = noOfPuts + noOfDeletes;
-this.metricsRegion.updatePut();
+//this.metricsRegion.updatePut();
}
if (noOfDeletes 0) {
// There were some Deletes in the batch.
-this.metricsRegion.updateDelete();
+//this.metricsRegion.updateDelete();
}
if (!success) {
for (int i = firstIndex; i lastIndexExclusive; i++) {
@@ -4270,7 +4270,7 @@ public class HRegion implements HeapSize { // ,
Writable{
// do after lock
-this.metricsRegion.updateGet();
+//this.metricsRegion.updateGet();
return results;
}
@@ -4657,7 +4657,7 @@ public class HRegion implements HeapSize { // ,
Writable{
closeRegionOperation();
}
-this.metricsRegion.updateAppend();
+//this.metricsRegion.updateAppend();
if (flush) {
@@ -4795,7 +4795,7 @@ public class HRegion implements HeapSize { // ,
Writable{
mvcc.completeMemstoreInsert(w);
}
closeRegionOperation();
- this.metricsRegion.updateIncrement();
+
[jira] [Commented] (HBASE-7225) on trunk, integration tests are not packaged into distribution
[ https://issues.apache.org/jira/browse/HBASE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506422#comment-13506422 ] Hudson commented on HBASE-7225: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #279 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/279/]) HBASE-7225 on trunk, integration tests are not packaged into distribution (Revision 1415056) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml * /hbase/trunk/src/assembly/components.xml on trunk, integration tests are not packaged into distribution -- Key: HBASE-7225 URL: https://issues.apache.org/jira/browse/HBASE-7225 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-7225.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6201) HBase integration/system tests
[
https://issues.apache.org/jira/browse/HBASE-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506421#comment-13506421
]
Hudson commented on HBASE-6201:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #279 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/279/])
HBASE-6201 Document how to run integration tests (Revision 1415055)
Result = FAILURE
stack :
Files :
* /hbase/trunk/src/docbkx/developer.xml
HBase integration/system tests
--
Key: HBASE-6201
URL: https://issues.apache.org/jira/browse/HBASE-6201
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Integration and general system tests have been discussed previously, and the
conclusion is that we need to unify how we do release candidate testing
(HBASE-6091).
In this issue, I would like to discuss and agree on a general plan, and open
subtickets for execution so that we can carry out most of the tests in
HBASE-6091 automatically.
Initially, here is what I have in mind:
1. Create hbase-it (or hbase-tests) containing forward port of HBASE-4454
(without any tests). This will allow integration test to be run with
{code}
mvn verify
{code}
2. Add ability to run all integration/system tests on a given cluster. Smt
like:
{code}
mvn verify -Dconf=/etc/hbase/conf/
{code}
should run the test suite on the given cluster. (Right now we can launch some
of the tests (TestAcidGuarantees) from command line). Most of the system
tests will be client side, and interface with the cluster through public
APIs. We need a tool on top of MiniHBaseCluster or improve
HBaseTestingUtility, so that tests can interface with the mini cluster or the
actual cluster uniformly.
3. Port candidate unit tests to the integration tests module. Some of the
candidates are:
- TestAcidGuarantees / TestAtomicOperation
- TestRegionBalancing (HBASE-6053)
- TestFullLogReconstruction
- TestMasterFailover
- TestImportExport
- TestMultiVersions / TestKeepDeletes
- TestFromClientSide
- TestShell and src/test/ruby
- TestRollingRestart
- Test**OnCluster
- Balancer tests
These tests should continue to be run as unit tests w/o any change in
semantics. However, given an actual cluster, they should use that, instead of
spinning a mini cluster.
4. Add more tests, especially, long running ingestion tests (goraci, BigTop's
TestLoadAndVerify, LoadTestTool), and chaos monkey style fault tests.
All suggestions welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7235) TestMasterObserver is flaky
[
https://issues.apache.org/jira/browse/HBASE-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506423#comment-13506423
]
Hudson commented on HBASE-7235:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #279 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/279/])
HBASE-7235 TestMasterObserver is flaky (Revision 1415005)
Result = FAILURE
enis :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java
TestMasterObserver is flaky
---
Key: HBASE-7235
URL: https://issues.apache.org/jira/browse/HBASE-7235
Project: HBase
Issue Type: Bug
Components: Coprocessors, master, test
Affects Versions: 0.96.0, 0.94.4
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.96.0, 0.94.4
Attachments: hbase-7235_v1.patch
TestMasterObserver failed on windows builds at least a couple of times,
although this is not windows specific.
Upon further inspection, it seems that we are affected by the race condition
for the table operations.
The handling of modify table occurs in its dedicated executor,
MASTER_TABLE_OPERATIONS, and happens in async, although the handling of
add/delete/modify columns are handled in the IPC handler thread. When we do
not wait for the first modify table to finish, subsequent changes from the
add/modify column operations are overriden by the earlier modify table
executor.
{code}
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.hasColumnFamily(TableEventHandler.java:182)
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.init(TableDeleteFamilyHandler.java:42)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.HBaseAdmin.deleteColumn(HBaseAdmin.java:1024)
at
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:616)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Meil updated HBASE-7221:
-
Attachment: hbase-common_hbase_7221_2.patch
Moved from server to common, renamed class name to RowKeyBuilder.
(7221_2.patch)
RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1648) heap usage and limit reporting doesn't work as expected when using G1 GC
[
https://issues.apache.org/jira/browse/HBASE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506547#comment-13506547
]
stack commented on HBASE-1648:
--
[~xieliang007] We should just close this then?
heap usage and limit reporting doesn't work as expected when using G1 GC
Key: HBASE-1648
URL: https://issues.apache.org/jira/browse/HBASE-1648
Project: HBase
Issue Type: Bug
Environment: Linux Centos x86_64 5.3, Sun JDK 1.6.0_14 (HotSpot
64-Bit Server VM build 14.0-b16)
Reporter: Andrew Purtell
Priority: Minor
Getting some early experience with the G1 GC.
Running with -Xmx1000m and HBASE_OPTS set to -XX:+UnlockExperimentalVMOpts
-XX:+UseG1GC. Regionserver heap use reports are not of much use:
{noformat}
hbase(main):001:0 status 'simple'
3 live servers
test3:60020 1247389283042
requests=0, regions=1, usedHeap=0, maxHeap=41
test2:60020 1247389219994
requests=0, regions=1, usedHeap=0, maxHeap=41
test4:60020 1247389324563
requests=0, regions=2, usedHeap=0, maxHeap=41
0 dead servers
{noformat}
top is about right:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
23988 hadoop24 0 1492m 98m 10m S 0.0 2.5 0:02.65 java
{noformat}
Incidentally, don't try -XX:+UseG1GC and -XX:+DoEscapeAnalysis together or
the JVM will rapidly segfault.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7120) hbase-daemon.sh (start) missing necessary check when writing pid and log files
[ https://issues.apache.org/jira/browse/HBASE-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506548#comment-13506548 ] stack commented on HBASE-7120: -- Agree that not having a pid will mess up our being able to stop the process. I think though that failing to write a pid is a reason sufficient not to start hbase. Can you store the pid in a variable in the start script and kill hbase if we fail writing the pid file? Similar w/ log files. If we can't write a log file, we should again fail the startup. What do you think [~michelle]? hbase-daemon.sh (start) missing necessary check when writing pid and log files -- Key: HBASE-7120 URL: https://issues.apache.org/jira/browse/HBASE-7120 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Environment: RHEL 5.3, open JDK 1.6 Reporter: Li Ping Zhang Assignee: Li Ping Zhang Labels: patch Attachments: HBASE-7120-0.94.patch, HBASE-7120-trunk.patch Original Estimate: 48h Remaining Estimate: 48h $HBASE_HOME/bin/hbase-daemon.sh exit code is Zero, when runing hbase-daemon.sh failed with start, which doesn’t do required command exit code check, it's better to do necessary check when writing pid and log files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[
https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506586#comment-13506586
]
Hadoop QA commented on HBASE-7215:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555249/7215v7.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3419//console
This message is automatically generated.
Put, Delete, Increment, and Result still implement Writable
---
Key: HBASE-7215
URL: https://issues.apache.org/jira/browse/HBASE-7215
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Blocker
Fix For: 0.96.0
Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt,
7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt,
MutableResult.java
Making blocker as suggested by Stack.
At least the following still use Put/Delete as writables.
* IdentityTableReduce.java
* MultiPut.java
* HRegionServer.checkAndMutate
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5968) Proper html escaping for region names
[
https://issues.apache.org/jira/browse/HBASE-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506598#comment-13506598
]
Elliott Clark commented on HBASE-5968:
--
Yep
Proper html escaping for region names
-
Key: HBASE-5968
URL: https://issues.apache.org/jira/browse/HBASE-5968
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.96.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
I noticed that we are not doing html escaping for the rs/master web
interfaces, so you can end up generating html like:
{code}
tr
tdci,,\xEEp/T\xBE\xC0,1336471826990.fc5a943e75ce8521b1ccdaf72d2c96c8./td
td
a href=hostnamehostname/a
/td
td,\xEEp/T\xBE\xC0/td
td-n\xA8\xE0\x15\xDD\x80!/td
td2966724/td
/tr
{code}
This obviously does not render properly.
Also, my crazy theory is that it can be a security risk. Since the region
name is computed from table rows, which are most of the time user input. Thus
if the rows contain a script onload= or similar, then that will be
executed on the developer's browser having possibly access to dev
environment.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506603#comment-13506603 ] Jimmy Xiang commented on HBASE-7204: Tool is more for MR. The patch still uses the regular configuration if it's not specified in system properties. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506639#comment-13506639 ] Jonathan Hsieh commented on HBASE-7204: --- Right -- what I'm saying is that I don't think we can specify the system property (as opposed to xml config property) if we use the normal hbase script. e.g. -- I don't think this will work as expected. hbase hbck -Dsystem.property=foo blah blah Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7225) on trunk, integration tests are not packaged into distribution
[ https://issues.apache.org/jira/browse/HBASE-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506642#comment-13506642 ] Sergey Shelukhin commented on HBASE-7225: - Thanks for all the commits :) on trunk, integration tests are not packaged into distribution -- Key: HBASE-7225 URL: https://issues.apache.org/jira/browse/HBASE-7225 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-7225.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506644#comment-13506644 ] Sergey Shelukhin commented on HBASE-7236: - Yes; the main difference is that default compaction config can have several parameters, and tiered compaction selection can have up to 20 (several parameters x few tier); this will make metadata hard to manage. So it might make sense to make it nested in some form. add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506655#comment-13506655 ] Jimmy Xiang commented on HBASE-7204: I tried it yesterday. We can run it like this: HBASE_OPTS=-Dkey=value hbase hbck ... Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506661#comment-13506661
]
stack commented on HBASE-7221:
--
Hey boss... fix the formatting. Its all over the place -- see the patch. You
have tabs in there? If you look at other code you'll see it uses spaces for
tabs and two spaces at that.
Builder is a good name but you don't seem to follow the general builder
pattern... i.e. get a 'builder', then do things against it and on the end call
'build' to return the result. That could confuse (You have some of it w/ your
static to create an instance...)
You keep adding to the backing array... just let the ArrayOutOfBounds happen if
they try to add off the end?
Why does the key have to be of fixed size?
Good stuff.
RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506662#comment-13506662 ] stack commented on HBASE-7215: -- I'm running hadoopqa manually. Rerunning again after the above inconclusive test result. Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506664#comment-13506664 ] Sergey Shelukhin commented on HBASE-7055: - The documentation for the selection algorithm is attached to HBASE-6371. Entire file is covered by this JIRA, except for most of 3.3. bq. Just do catch (Exception e) once? Hmm... strong reason to do so? The code is kind of verbose this way, but it catches only what it intends to catch. bq. What is the min flush time about? This is used as the file time for the purposes of assigning files to tiers based on time. bq. Is 'TierCompaction' the default as it says in the class comment: '+ * Control knobs for default compaction algorithm.'? No; changed comment. bq. Why the break out of config for TierCompaction in particular? Will we have to do this pattern for all we'd dynamically config: i.e. break out a Config class when we are already carrying a heavyweight Configuration anyways that is mostly accessible from anywhere? Do you mean Configuration or CompactionConfiguration by large object? CompactionConfiguration is base compaction config, it is not just xml-based, it uses runtime store-specific settings. TierBased one adds more on top of that; it seems that Tier-stuff doesn't belong to the main CompactionConfiguration; and main CompactionConfiguration is not as simple as generic Configuration. It's also Store (e.g. region/cf) specific. bq. I wonder who is going to take the time to do configuration on each compaction tier? Is this asking a bit much of operators? The doc in HBASE-6371 shows examples of separate configuration for tiers; initial scenario for this may have been to compact middle files more aggressively than either old or recent files, so that would require tier tweaking. Old compaction selection policy is on by default so the operators needn't worry :) bq. There is no class comment on TierCompactionPolicy, the place I'd go to learn about how TierCompactionPolicy works. That is an interesting question... there's an exhaustive doc on it, should it be copied to book.xml and referenced in javadoc, or summarized? bq. Does this policy do anything about making it so leveldb-like, every file may contain all keys in the namespace: i.e. does it make it so a tier is made of files that each contain a distinct subset of the namespace? No, the similarity in names is misleading, they don't have a lot in common. port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7055: Attachment: HBASE-7055-v1.patch Patch comment: comments, whitespace, file renames port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506689#comment-13506689 ] Jonathan Hsieh commented on HBASE-7204: --- Ok, well that needs to be documented (release notes?). Honestly, I'd still prefer if this was turned into a Tool since the rest of the hbase tools like import/export/copytable/LoadIncrementalHFiles do this. I think this is a hadoop platform convention, not just an MR convention. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506695#comment-13506695 ] Jimmy Xiang commented on HBASE-7204: Cool. Can we handle that in a separate jira so that we can have this capability now (overriding this conf from command line for hbck)? Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[
https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506700#comment-13506700
]
Jonathan Hsieh commented on HBASE-7204:
---
I'm -0 on the current approach (not going to block but don't like), +1 if tool
is used.
Using Tool is really simple -- just rename main to run and replace main with
the equivalent of this.
{code}
public static void main(String[] args) throws Exception {
int ret = ToolRunner.run(new
LoadIncrementalHFiles(HBaseConfiguration.create()), args);
System.exit(ret);
}
{code}
Make hbck ErrorReporter pluggable
-
Key: HBASE-7204
URL: https://issues.apache.org/jira/browse/HBASE-7204
Project: HBase
Issue Type: Improvement
Components: hbck
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-7204.patch
Make hbck ErrorReporter pluggable so that it can be replaced dynamically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[
https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506704#comment-13506704
]
Hadoop QA commented on HBASE-7215:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555249/7215v7.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 25 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3420//console
This message is automatically generated.
Put, Delete, Increment, and Result still implement Writable
---
Key: HBASE-7215
URL: https://issues.apache.org/jira/browse/HBASE-7215
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Blocker
Fix For: 0.96.0
Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt,
7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt,
MutableResult.java
Making blocker as suggested by Stack.
At least the following still use Put/Delete as writables.
* IdentityTableReduce.java
* MultiPut.java
* HRegionServer.checkAndMutate
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506707#comment-13506707
]
Elliott Clark commented on HBASE-4709:
--
My thought was putting the defaults in the log4j conf + in both of the
MetricsAssertHelpers.
Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
Key: HBASE-4709
URL: https://issues.apache.org/jira/browse/HBASE-4709
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0, 0.94.0, 0.94.1
Reporter: Gary Helmling
Priority: Minor
Fix For: 0.96.0
Attachments: 4709_workaround.v1.patch
Since switching over HBase to build with Hadoop 0.20.205.0, we've been
getting a lot of metrics related errors in the log files for tests:
{noformat}
2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
SelectChannelConnector@localhost:55751
Starting DataNode 1 with dfs.data.dir:
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
system not started: Cannot locate configuration: tried
hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered
with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at
com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at
com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at
com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
{noformat}
This seems to be due to errors initializing the new hadoop metrics2 code by
default, when running in a mini cluster. The errors themselves seem to be
harmless -- they're not breaking any tests -- but we should figure out what
configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7237) system metadata for tables/cfs needs to be validated on the master
Sergey Shelukhin created HBASE-7237: --- Summary: system metadata for tables/cfs needs to be validated on the master Key: HBASE-7237 URL: https://issues.apache.org/jira/browse/HBASE-7237 Project: HBase Issue Type: Improvement Reporter: Sergey Shelukhin Priority: Minor shell currently validates what used provides with alter, however while looking at some other issue I noticed that user and system metadata is stored in the same dictionary on the server, so it is easy to bypass by setting a user metadata parameter with the same name as the system parameter. E.g. I just set MAX_FILESIZE to moo via CONFIG. This can be fixed in the shell, however the general problem I think is that system configuration should be validated server-side (e.g. on the master), not just on the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7237) system metadata for tables/cfs needs to be validated on the master
[ https://issues.apache.org/jira/browse/HBASE-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7237: Description: shell currently validates whatever metadata user provides as argument to alter, however while looking at some other issue I noticed that user and system metadata is stored in the same dictionary in the descriptor, so shell validation is easy to bypass by setting a user metadata parameter with the same name as the system parameter. E.g. I just set MAX_FILESIZE to moo via CONFIG. This can be fixed in the shell, however the general problem I think is that system configuration should be validated server-side (e.g. on the master), not just on the client. was: shell currently validates what used provides with alter, however while looking at some other issue I noticed that user and system metadata is stored in the same dictionary on the server, so it is easy to bypass by setting a user metadata parameter with the same name as the system parameter. E.g. I just set MAX_FILESIZE to moo via CONFIG. This can be fixed in the shell, however the general problem I think is that system configuration should be validated server-side (e.g. on the master), not just on the client. system metadata for tables/cfs needs to be validated on the master -- Key: HBASE-7237 URL: https://issues.apache.org/jira/browse/HBASE-7237 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Priority: Minor shell currently validates whatever metadata user provides as argument to alter, however while looking at some other issue I noticed that user and system metadata is stored in the same dictionary in the descriptor, so shell validation is easy to bypass by setting a user metadata parameter with the same name as the system parameter. E.g. I just set MAX_FILESIZE to moo via CONFIG. This can be fixed in the shell, however the general problem I think is that system configuration should be validated server-side (e.g. on the master), not just on the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506711#comment-13506711 ] Jimmy Xiang commented on HBASE-7204: Okay. Let me convert it to Tool and post a new patch. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7237) system metadata for tables/cfs needs to be validated on the master
[ https://issues.apache.org/jira/browse/HBASE-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7237: Affects Version/s: 0.96.0 system metadata for tables/cfs needs to be validated on the master -- Key: HBASE-7237 URL: https://issues.apache.org/jira/browse/HBASE-7237 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Priority: Minor shell currently validates whatever metadata user provides as argument to alter, however while looking at some other issue I noticed that user and system metadata is stored in the same dictionary in the descriptor, so shell validation is easy to bypass by setting a user metadata parameter with the same name as the system parameter. E.g. I just set MAX_FILESIZE to moo via CONFIG. This can be fixed in the shell, however the general problem I think is that system configuration should be validated server-side (e.g. on the master), not just on the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Status: Open (was: Patch Available) Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506720#comment-13506720 ] Lars Hofhansl commented on HBASE-7215: -- That looks pretty good! Should probably still upload to RB for easier review. Two sticky points I know about: # Result deserialization used to do 8k chunked reading from the input stream to avoid OOM'ing on the JVMs direct memory. I have no idea how protobufs reads from the stream when deserializing so we may get that problem back. # The size metric in ScannerCallable was broken when protobufs were introduced, because that just measured the size of the bytes stream (that is no longer filled with protobufs). This patch just comments that part, because it was broken anyway. Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506721#comment-13506721 ] stack commented on HBASE-7215: -- +1 on commit Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506730#comment-13506730 ] Todd Lipcon commented on HBASE-7213: I agree it seems to make sense to lump this with the multi-WAL work. Perhaps an interface like WALFactory or WALProvider, which, given a region name, gives back a WAL instance? The basic implementation would always provide the single WAL. Then, we could add the feature that returns a different WAL for META alone. More complex implementations could choose to give different tenants of a cluster separate WALs, etc. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506733#comment-13506733 ] stack commented on HBASE-7215: -- Make new issues for the two probs above? Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7238) Size based scan metric broken by protobufs
Lars Hofhansl created HBASE-7238: Summary: Size based scan metric broken by protobufs Key: HBASE-7238 URL: https://issues.apache.org/jira/browse/HBASE-7238 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Fix For: 0.96.0 See ScannerCallable. HBASE-7215 comments that portion, but it did not work before, because Results.bytes is no longer used with protobufs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7239) Verify protobuf serialization is correctly chunking upon read to avoid direct memory OOMs
Lars Hofhansl created HBASE-7239: Summary: Verify protobuf serialization is correctly chunking upon read to avoid direct memory OOMs Key: HBASE-7239 URL: https://issues.apache.org/jira/browse/HBASE-7239 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Result.readFields() used to read from the input stream in 8k chunks to avoid OOM issues with direct memory. (Reading variable sized chunks into direct memory prevent the JVM from reusing the allocated direct memory and direct memory is only collected during full GCs) This is just to verify protobufs parseFrom type methods do the right thing as well so that we do not reintroduce this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506739#comment-13506739 ] Lars Hofhansl commented on HBASE-7215: -- Filed two sub tasks. OK will commit in the next hour without an extra RB review unless I hear objections. Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7239) Verify protobuf serialization is correctly chunking upon read to avoid direct memory OOMs
[ https://issues.apache.org/jira/browse/HBASE-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7239: - Priority: Critical (was: Major) Verify protobuf serialization is correctly chunking upon read to avoid direct memory OOMs - Key: HBASE-7239 URL: https://issues.apache.org/jira/browse/HBASE-7239 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Priority: Critical Fix For: 0.96.0 Result.readFields() used to read from the input stream in 8k chunks to avoid OOM issues with direct memory. (Reading variable sized chunks into direct memory prevent the JVM from reusing the allocated direct memory and direct memory is only collected during full GCs) This is just to verify protobufs parseFrom type methods do the right thing as well so that we do not reintroduce this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7238) Size based scan metric broken by protobufs
[ https://issues.apache.org/jira/browse/HBASE-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7238: - Priority: Critical (was: Major) Size based scan metric broken by protobufs -- Key: HBASE-7238 URL: https://issues.apache.org/jira/browse/HBASE-7238 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Priority: Critical Fix For: 0.96.0 See ScannerCallable. HBASE-7215 comments that portion, but it did not work before, because Results.bytes is no longer used with protobufs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7124) typo in pom.xml with exlude, no definition of test.exclude.pattern
[ https://issues.apache.org/jira/browse/HBASE-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506755#comment-13506755 ] Jesse Yates commented on HBASE-7124: Looks good to me. I'll commit today, unless there are any objections. typo in pom.xml with exlude, no definition of test.exclude.pattern -- Key: HBASE-7124 URL: https://issues.apache.org/jira/browse/HBASE-7124 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Li Ping Zhang Assignee: Li Ping Zhang Priority: Minor Labels: patch Attachments: HBASE-7124-0.94.patch, HBASE-7124-v1.patch Original Estimate: 4h Remaining Estimate: 4h There is a typo in pom.xml with exlude, and there is no definition of test.exclude.pattern. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506761#comment-13506761 ] Andrew Purtell commented on HBASE-7236: --- Encode compaction selection as JSON in the attribute I'd say. add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506795#comment-13506795 ] Andrew Purtell commented on HBASE-7055: --- bq. CompactionConfiguration is base compaction config, it is not just xml-based, it uses runtime store-specific settings. TierBased one adds more on top of that; it seems that Tier-stuff doesn't belong to the main CompactionConfiguration; and main CompactionConfiguration is not as simple as generic Configuration. It's also Store (e.g. region/cf) specific. Up to now we've had two distinct and cleanly separable configuration mechanisms. The heavyweight Configuration which carries global, and currently static configuration, and the table and column descriptors that can be CF specific by definition and updated at runtime without requiring a process restart. Pardon if I've misunderstood but mixing these would blur static and dynamic configuration and that doesn't seem a good design option. port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7232: - Status: Open (was: Patch Available) Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7232: - Status: Patch Available (was: Open) Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7232: - Attachment: 7232v2.txt Cleaned out more Writables from hfile package. Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5926) Delete the master znode after a master crash
[
https://issues.apache.org/jira/browse/HBASE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506801#comment-13506801
]
Jean-Daniel Cryans commented on HBASE-5926:
---
This jira has the odd side-effect of printing out a lot of garbage when running
in standalone and killing it with -9, gist of it being:
{noformat}
2012-11-29 13:08:27,227 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase/master
2012-11-29 13:08:27,227 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper getData
failed after 0 retries
2012-11-29 13:08:27,227 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: clean
znode for master Unable to get data of znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /hbase/master
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:291)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:562)
at
org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.deleteIfEquals(MasterAddressTracker.java:168)
at org.apache.hadoop.hbase.ZNodeClearer.clear(ZNodeClearer.java:150)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:110)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:78)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2298)
{noformat}
Basically the znode cleaner fails hard because ZK is offline.
I was confused to see more logs being printed out after running the kill.
Delete the master znode after a master crash
Key: HBASE-5926
URL: https://issues.apache.org/jira/browse/HBASE-5926
Project: HBase
Issue Type: Improvement
Components: master, scripts
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
Attachments: 5926.v10.patch, 5926.v11.patch, 5926.v13.patch,
5926.v14.patch, 5926.v6.patch, 5926.v8.patch, 5926.v9.patch
This is the continuation of the work done in HBASE-5844.
But we can't apply exactly the same strategy: for the region server, there is
a znode per region server, while for the master backup master there is a
single znode for both.
So if we apply the same strategy as for a regionserver, we may have this
scenario:
1) Master starts
2) Backup master starts
3) Master dies
4) ZK detects it
5) Backup master receives the update from ZK
6) Backup master creates the new master node and become the main master
7) Previous master script continues
8) Previous master script deletes the master node in ZK
9) = issue: we deleted the node just created by the new master
This should not happen often (usually the znode will be deleted soon enough),
but it can happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7240) Cleanup old snapshots on start
Jesse Yates created HBASE-7240: -- Summary: Cleanup old snapshots on start Key: HBASE-7240 URL: https://issues.apache.org/jira/browse/HBASE-7240 Project: HBase Issue Type: Sub-task Affects Versions: hbase-6055 Reporter: Jesse Yates Fix For: hbase-6055 If the master is hard stopped (i.e. kill -9), the snapshot handler or SnapshotManager may not have a chance to cleanup after the snapshot, leaving extraneous files in the working snapshot directory (/hbase/.snapshot/.tmp directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7240) Cleanup old snapshots on start
[ https://issues.apache.org/jira/browse/HBASE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506808#comment-13506808 ] Jesse Yates commented on HBASE-7240: We could create a background chore that periodically checks this directory against the running snapshots, but its likely a rare occurrence that we fail a snapshot and can't cleanup after it. We probably just need to add a little cleanup mechanism to on startup (could just drop this into the SnapshotManager as it also plays well into a possible future goal of recovering snapshot attempts between master failures). Cleanup old snapshots on start -- Key: HBASE-7240 URL: https://issues.apache.org/jira/browse/HBASE-7240 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Fix For: hbase-6055 If the master is hard stopped (i.e. kill -9), the snapshot handler or SnapshotManager may not have a chance to cleanup after the snapshot, leaving extraneous files in the working snapshot directory (/hbase/.snapshot/.tmp directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506812#comment-13506812
]
Doug Meil commented on HBASE-7221:
--
re: Builder
Yeah, I really wasn't going for a builder pattern. Elliott had a concern about
the name RowKey (I must admit I'm still partial to it because there isn't a
class with that name anywhere in the codebase).
I wasn't really aiming for a builder pattern in the first place because I
didn't want to necessarily force people to destroy and re-create the
RowKey/Builder for each rowkey they create - that's why the reset method is
there. The only thing that would have to get reset was the backing byte array.
re: fixed size
I wanted any particular instance to have a fixed size so that the backing
byte-array didn't have resize like an ArrayList (and wind up burning a lot of
byte-arrays in the process). So it's easier to create rowkeys than without
the utility, but not without required thought.
If your table had multiple length keys, there's nothing wrong with creating 2
different instances, one for each length.
That's where I was coming from.
re: formatting
I'll fix that. Doh!
Thanks!
RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5844) Delete the region servers znode after a regions server crash
[
https://issues.apache.org/jira/browse/HBASE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506832#comment-13506832
]
Jean-Daniel Cryans commented on HBASE-5844:
---
Encountered another problem that I think I can link to this jira, I was trying
to run HBase from trunk without internet access and like in my Sept 25th
comment, I get an empty line after start-hbase.sh but now nothing is running.
The .log file doesn't show anything after logging ulimit and nothing's in the
.out file. After running some bash -x, I was able to figure out that the nohup
output was being suppressed. See:
{noformat}
jdcryans-MBPr:hbase-github jdcryans$ ./bin/start-hbase.sh
jdcryans-MBPr:hbase-github jdcryans$
jdcryans-MBPr:hbase-github jdcryans$ bash -x ./bin/start-hbase.sh
... some stuff then
+ /Users/jdcryans/git/hbase-github/bin/hbase-daemon.sh start master
jdcryans-MBPr:hbase-github jdcryans$ bash -x
/Users/jdcryans/git/hbase-github/bin/hbase-daemon.sh start master
... more stuff
+ nohup /Users/jdcryans/git/hbase-github/bin/hbase-daemon.sh --config
/Users/jdcryans/git/hbase-github/bin/../conf internal_start master
jdcryans-MBPr:hbase-github jdcryans$ nohup
/Users/jdcryans/git/hbase-github/bin/hbase-daemon.sh --config
/Users/jdcryans/git/hbase-github/bin/../conf internal_start master
appending output to nohup.out
{noformat}
So now I see that it's writing to nohup.out, which in turn tells me what really
happened:
{noformat}
Caused by: java.lang.ClassNotFoundException:
org.apache.zookeeper.KeeperException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
{noformat}
Reproing can be done by physically deleting any jar listed in
target/cached_classpath.txt. In my case I think the jar wasn't available
because I had no internet connection.
I wonder what other errors it could hide like this.
Delete the region servers znode after a regions server crash
Key: HBASE-5844
URL: https://issues.apache.org/jira/browse/HBASE-5844
Project: HBase
Issue Type: Improvement
Components: regionserver, scripts
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Fix For: 0.96.0
Attachments: 5844.v1.patch, 5844.v2.patch, 5844.v3.patch,
5844.v3.patch, 5844.v4.patch
today, if the regions server crashes, its znode is not deleted in ZooKeeper.
So the recovery process will stop only after a timeout, usually 30s.
By deleting the znode in start script, we remove this delay and the recovery
starts immediately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506848#comment-13506848
]
Doug Meil commented on HBASE-7221:
--
One more thought on size: then again, I could do what ArrayList does with it's
overloaded constructor - use that size initially, and then auto-size if needed.
But you could still define the exact size if you wanted for performance
purposes. that's probably the nicest possible approach.
RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506850#comment-13506850 ] Sergey Shelukhin commented on HBASE-7055: - It doesn't read CF values directly, what it does as encapsulate the logic such as runtime-based defaults (e.g. memstore flush size for min compaction size, throttling default, etc.) and some validation. This logic was already there, now it's just in separate class. For tiered one, also getting the requisite config for tiers based on tier count. Also, as an aside, configs are already mixed inside HStore (see CompoundConfiguration creation in ctor). port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Attachment: trunk-7204_v2.patch Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Release Note: Now hbck runs with ToolRunner, and can accept configurations from command line. Status: Patch Available (was: Open) Addressed Jon and Ram's comments. Now hbck runs with ToolRunner, and can accept configurations from command line. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506873#comment-13506873 ] Jonathan Hsieh commented on HBASE-7204: --- +1 lgtm. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Summary: Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable (was: Put, Delete, Increment, and Result still implement Writable) Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable -- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506877#comment-13506877 ] Jimmy Xiang commented on HBASE-7204: Thanks for the review. I need to do a minor change to fix a test failure. I will do that before I commit. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the help and review Stack. Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable -- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506884#comment-13506884 ] Ted Yu commented on HBASE-7213: --- We already have HLogFactory in trunk. bq. to lump this with the multi-WAL work Can I interpret the above as saying that multi-WAL work should be done at the same time, if not earlier ? Since HLogFactory can hand out the unique instance for .META., it is not far from handing out different instances (for different regions) which is what HBASE-5699 tries to do. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506885#comment-13506885 ] Jonathan Hsieh commented on HBASE-7204: --- Thanks for making the changes! feel free to commit if the test fix is trivial Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
Doug Meil created HBASE-7241: Summary: [refGuide] Update to Performannce/writing/presplit Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
[ https://issues.apache.org/jira/browse/HBASE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-7241: - Attachment: docbkx_hbase_7241.patch [refGuide] Update to Performannce/writing/presplit -- Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_7241.patch Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
[ https://issues.apache.org/jira/browse/HBASE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-7241: - Status: Patch Available (was: Open) [refGuide] Update to Performannce/writing/presplit -- Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_7241.patch Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
[ https://issues.apache.org/jira/browse/HBASE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-7241: - Resolution: Fixed Status: Resolved (was: Patch Available) [refGuide] Update to Performannce/writing/presplit -- Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_7241.patch Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506907#comment-13506907 ] Elliott Clark commented on HBASE-7232: -- * In HBaseObjectWritable is it cleaner to just increment the code (like on line 258) rather than putting Object in the map ? * Would having separate implementations of the HFile.FileInfo with different reader methods be worth it ? ( hfilev1 and hvile =v2.1 would have the writable. Everything else uses the pb. Could make removing writable version easier later) * HFileWriterV2 is a white space only change is that intended ? Some thoughts about code this patch happens to touch: * Seems like most of the CompoundBloomFilter classes belong in io. Worth moving them now ? * Should CompoundBloomFilterWriter#cacheOnWrite() be renamed to getCacheOnWrite ? Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506923#comment-13506923 ] Devaraj Das commented on HBASE-7213: [~tlipcon], good points there. But I'd like to separate the META SPOF work from the full fledged multiwal work (for the full multi-wal case, we'd need to fix up things like replication, and those can be skipped from the meta-only design that this jira attempts to do). [~yuzhih...@gmail.com], I guess you are right. In theory, one could read the HLogFactory as the WALFactory... Thoughts? (i am in the process of extending the patch I previously posted to a fully functional one) Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5877) When a query fails because the region has moved, let the regionserver return the new address to the client
[
https://issues.apache.org/jira/browse/HBASE-5877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506935#comment-13506935
]
Jean-Daniel Cryans commented on HBASE-5877:
---
Nicolas,
Do you think this log message could be removed?
{noformat}
12/11/29 15:17:36 INFO client.HConnectionManager$HConnectionImplementation:
Region TestTable,0001966229,1354231005211.1bbba78dda968874d2981c322ed3319f.
moved from 572ba57e-1cab-4f9c-a071-782e5a1a7184.cs1cloud.internal:60020,
updating client location cache. New server:
20590793-0e19-4eb4-b2f6-05de8244f716.cs1cloud.internal:60020
{noformat}
Right now I'm running some loading tests and I'm getting walls of text every
time a split happens and it's basically the same message repeated hundreds of
times. We used to have a similar message before but we removed it since it's
pretty spammy (or we set it to DEBUG, can't remember).
When a query fails because the region has moved, let the regionserver return
the new address to the client
--
Key: HBASE-5877
URL: https://issues.apache.org/jira/browse/HBASE-5877
Project: HBase
Issue Type: Improvement
Components: Client, master, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Fix For: 0.96.0
Attachments: 5877.v12.patch, 5877.v15.patch, 5877-v16.txt,
5877-v17.txt, 5877-v17.txt, 5877.v18.patch, 5877.v18.patch, 5877.v18.patch,
5877.v1.patch, 5877.v6.patch
This is mainly useful when we do a rolling restart. This will decrease the
load on the master and the network load.
Note that a region is not immediately opened after a close. So:
- it seems preferable to wait before retrying on the other server. An
optimisation would be to have an heuristic depending on when the region was
closed.
- during a rolling restart, the server moves the regions then stops. So we
may have failures when the server is stopped, and this patch won't help.
The implementation in the first patch does:
- on the region move, there is an added parameter on the regionserver#close
to say where we are sending the region
- the regionserver keeps a list of what was moved. Each entry is kept 100
seconds.
- the regionserver sends a specific exception when it receives a query on a
moved region. This exception contains the new address.
- the client analyses the exeptions and update its cache accordingly...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7240) Cleanup old snapshots on start
[ https://issues.apache.org/jira/browse/HBASE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506939#comment-13506939 ] Matteo Bertozzi commented on HBASE-7240: +1 on just have the cleanup at startup Cleanup old snapshots on start -- Key: HBASE-7240 URL: https://issues.apache.org/jira/browse/HBASE-7240 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Fix For: hbase-6055 If the master is hard stopped (i.e. kill -9), the snapshot handler or SnapshotManager may not have a chance to cleanup after the snapshot, leaving extraneous files in the working snapshot directory (/hbase/.snapshot/.tmp directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506940#comment-13506940 ] Jeffrey Zhong commented on HBASE-6367: -- Starting to work on this one. List backup masters in ui. -- Key: HBASE-6367 URL: https://issues.apache.org/jira/browse/HBASE-6367 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Priority: Minor Labels: noob Right now only the active master shows any information on the web ui. It would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6052) Convert .META. and -ROOT- content to pb
[
https://issues.apache.org/jira/browse/HBASE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506951#comment-13506951
]
Jean-Daniel Cryans commented on HBASE-6052:
---
Enis,
Can we remove this log message?
{code}
public static HRegionInfo getHRegionInfo(Result data) {
...
if (LOG.isDebugEnabled()) {
LOG.debug(Current INFO from scan results = + info);
}
return info;
}
{code}
It's not too spammy if you have a few regions but as soon as you reach the
hundreds it becomes quite excessive. For example, try running this:
bq. bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --presplit=500
randomWrite 5
Convert .META. and -ROOT- content to pb
---
Key: HBASE-6052
URL: https://issues.apache.org/jira/browse/HBASE-6052
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: Enis Soztutar
Priority: Blocker
Fix For: 0.96.0
Attachments: 6052-v5.txt, 6052_v8.patch, HBASE-6052_v1.patch,
HBASE-6052_v2.patch, HBASE-6052_v3.patch, HBASE-6052_v4.patch,
HBASE-6052_v4.patch, HBASE-6052_v7.patch, HBASE-6052_v8.patch,
HBASE-6052_v9.patch, TestMetaMigrationConvertToPB.tgz
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506953#comment-13506953 ] Todd Lipcon commented on HBASE-7213: Yea, I don't think you need to strictly sequence this after the multi-WAL work. But it would be nice to have the end goal in mind while doing this work. Sorry, haven't had time to look at the in-progress patch, but if there's a simple solution that works OK now, no sense blocking it for the perfect end-game solution later. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506961#comment-13506961 ] Hudson commented on HBASE-7215: --- Integrated in HBase-TRUNK #3580 (See [https://builds.apache.org/job/HBase-TRUNK/3580/]) HBASE-7215 Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable (Revision 1415412) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Action.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Append.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Get.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Increment.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPut.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPutResponse.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/OperationWithAttributes.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableReduce.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableReducer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MutationSerialization.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ResultSerialization.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableReducer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable -- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
[ https://issues.apache.org/jira/browse/HBASE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506962#comment-13506962 ] Hudson commented on HBASE-7241: --- Integrated in HBase-TRUNK #3580 (See [https://builds.apache.org/job/HBase-TRUNK/3580/]) hbase-7241. refGuide. Perf/Schema design cleanup. (Revision 1415422) Result = FAILURE [refGuide] Update to Performannce/writing/presplit -- Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_7241.patch Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Attachment: 0.94-7204.patch Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 0.94-7204.patch, trunk-7204.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Status: Open (was: Patch Available) Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 0.94-7204.patch, trunk-7204.patch, trunk-7204_v2.1.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Attachment: trunk-7204_v2.1.patch Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 0.94-7204.patch, trunk-7204.patch, trunk-7204_v2.1.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7204: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) Try hadoopqa again. Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: 0.94-7204.patch, trunk-7204.patch, trunk-7204_v2.1.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506969#comment-13506969 ] Hudson commented on HBASE-7215: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #280 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/280/]) HBASE-7215 Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable (Revision 1415412) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-examples/src/main/java/org/apache/hadoop/hbase/mapreduce/IndexBuilder.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Action.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Append.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Get.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Increment.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiAction.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPut.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPutResponse.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiResponse.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/OperationWithAttributes.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Result.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMap.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableReduce.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableReducer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MutationSerialization.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ResultSerialization.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableReducer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAttributes.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTimestampsFilter.java Put, Delete, Increment, Result, all all HBase M/R classes still implement/use Writable -- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Attachments: 7215-v2.txt, 7215v3_mutableresult.txt, 7215v3.txt, 7215v4.txt, 7215v5.txt, 7215v6.txt, 7215v7.txt, 7215v7.txt, 7251-SKETCH.txt, MutableResult.java Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7241) [refGuide] Update to Performannce/writing/presplit
[ https://issues.apache.org/jira/browse/HBASE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506970#comment-13506970 ] Hudson commented on HBASE-7241: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #280 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/280/]) hbase-7241. refGuide. Perf/Schema design cleanup. (Revision 1415422) Result = FAILURE [refGuide] Update to Performannce/writing/presplit -- Key: HBASE-7241 URL: https://issues.apache.org/jira/browse/HBASE-7241 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_7241.patch Took the pre-split example that was in Performance/Writing/Pre-split and moved it to the Schema Design/RowKey-PreSplit section that was just created. Updated the Perf/pre-split section to additionally refer to the new RowKey-PreSplit section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7242) Use Runtime.exit() instead of Runtime.halt() upon HLog flush failures
Amitanand Aiyer created HBASE-7242: -- Summary: Use Runtime.exit() instead of Runtime.halt() upon HLog flush failures Key: HBASE-7242 URL: https://issues.apache.org/jira/browse/HBASE-7242 Project: HBase Issue Type: Brainstorming Reporter: Amitanand Aiyer Priority: Minor Hey Guys, Should we use Runtime.exit() instead of Runtime.halt(), when we fail a Hlog sync. The key difference is that Runtime.exit() is going to invoke the shutdown hooks; while Runtime.halt() does not. Why we might need this: We had a HDFS name node reboot today on one of our cells, and this caused multiple region servers to abort because they could not sync the Hlog. However, since multiple RS died simultaneously, this seemed like a co-related failure to the master. The master waits for the Znode to expire; but, this could take up to few minutes after RS death (this setting is in place so that we can withstand rack switch reboots, lasting a couple of minutes, without region movement). If the shutdown hooks are called, RS will close the ZK connection, causing a immediate Znode expiry. This might help cut down the unavailability as Regions can begin to get assigned faster. While, we do want to abort on Hlog failure, I do not think it would hurt giving the JVM a few seconds to shutdown gracefully. Please let me know If I am missing something. Thanks, -Amit -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7242) Use Runtime.exit() instead of Runtime.halt() upon HLog Sync failures
[ https://issues.apache.org/jira/browse/HBASE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-7242: --- Summary: Use Runtime.exit() instead of Runtime.halt() upon HLog Sync failures (was: Use Runtime.exit() instead of Runtime.halt() upon HLog flush failures) Use Runtime.exit() instead of Runtime.halt() upon HLog Sync failures Key: HBASE-7242 URL: https://issues.apache.org/jira/browse/HBASE-7242 Project: HBase Issue Type: Brainstorming Reporter: Amitanand Aiyer Priority: Minor Hey Guys, Should we use Runtime.exit() instead of Runtime.halt(), when we fail a Hlog sync. The key difference is that Runtime.exit() is going to invoke the shutdown hooks; while Runtime.halt() does not. Why we might need this: We had a HDFS name node reboot today on one of our cells, and this caused multiple region servers to abort because they could not sync the Hlog. However, since multiple RS died simultaneously, this seemed like a co-related failure to the master. The master waits for the Znode to expire; but, this could take up to few minutes after RS death (this setting is in place so that we can withstand rack switch reboots, lasting a couple of minutes, without region movement). If the shutdown hooks are called, RS will close the ZK connection, causing a immediate Znode expiry. This might help cut down the unavailability as Regions can begin to get assigned faster. While, we do want to abort on Hlog failure, I do not think it would hurt giving the JVM a few seconds to shutdown gracefully. Please let me know If I am missing something. Thanks, -Amit -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7219) Make it so can connect remotely to a standalone hbase
[ https://issues.apache.org/jira/browse/HBASE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506974#comment-13506974 ] Jean-Daniel Cryans commented on HBASE-7219: --- bq. HBase has 'localhost' in regionservers file This doesn't have anything to do with how we report addresses tho, it's only used for ssh. bq. and will write 'localhost' to znode for master location which remote client can't use AFAIK we don't force localhost in there, what we do force is looking up localhost for the zookeeper server by default and even then a remote client could just specify the right address and it should work. If localhost is reported for the master or RS, it's a problem with how the node identifies itself. Make it so can connect remotely to a standalone hbase - Key: HBASE-7219 URL: https://issues.apache.org/jira/browse/HBASE-7219 Project: HBase Issue Type: Bug Reporter: stack Should be able to connect from a remote client to a standalone instance. HBase has 'localhost' in regionservers file and will write 'localhost' to znode for master location which remote client can't use. Fix. This comes up on mailing list w/ some frequency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7236: Attachment: HBASE-7236-PROTOTYPE.patch After poking around a bit, I think it's a good idea to have direct config key override. Justification is that having separately names metadata fields will be hard to manage given the potential number of fields. Ditto for JSON serialization - it would require special logic to validate/use overrides, and shell support would be painful. Perhaps we can white-list configs that can be overridden this way, too, inside HTableDescriptor. Here's the prototype for table level only, and with no shell support and tests for now. Technically, this approach already works in Store for column family (CompoundConfiguration is created with family metadata overriding Configuration, so if someone adds a key with correct name to family metadata it will override xml config within Store); however, making it explicit and separate from miscellaneous metadata would be cleaner imho. Please comment... Thanks! add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7236-PROTOTYPE.patch Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7234: - Attachment: 7234.txt Remove long-deprecated HServerAddress and HServerInfo Writables --- Key: HBASE-7234 URL: https://issues.apache.org/jira/browse/HBASE-7234 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Attachments: 7234.txt These classes have been deprecated since 0.92 or before. Remove them. Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506984#comment-13506984 ] stack commented on HBASE-7232: -- bq. In HBaseObjectWritable is it cleaner to just increment the code (like on line 258) rather than putting Object in the map ? Yes. Thanks. bq. Would having separate implementations of the HFile.FileInfo with different reader methods be worth it ? More pain than it is worth IMO. bq. HFileWriterV2 is a white space only change is that intended ? Let me remove from the next revision. bq. Seems like most of the CompoundBloomFilter classes belong in io. Worth moving them now ? Not as part of this patch I'd say. They need a bit of work to undo Writables. Might mess up backward compatibility moving their location. Would need to check if class name is written to the hfile). bq. Should CompoundBloomFilterWriter#cacheOnWrite() be renamed to getCacheOnWrite ? Yes. Thanks for the review. Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7234) Remove long-deprecated HServerAddress and HServerInfo Writables
[ https://issues.apache.org/jira/browse/HBASE-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7234: - Fix Version/s: 0.96.0 Status: Patch Available (was: Open) Remove long-deprecated HServerAddress and HServerInfo Writables --- Key: HBASE-7234 URL: https://issues.apache.org/jira/browse/HBASE-7234 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7234.txt These classes have been deprecated since 0.92 or before. Remove them. Remove them too because they are Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7240) Cleanup old snapshots on start
[ https://issues.apache.org/jira/browse/HBASE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506989#comment-13506989 ] Ted Yu commented on HBASE-7240: --- Cleanup at startup would be completed quickly, right ? I assume this wouldn't affect total recovery time much. Cleanup old snapshots on start -- Key: HBASE-7240 URL: https://issues.apache.org/jira/browse/HBASE-7240 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Fix For: hbase-6055 If the master is hard stopped (i.e. kill -9), the snapshot handler or SnapshotManager may not have a chance to cleanup after the snapshot, leaving extraneous files in the working snapshot directory (/hbase/.snapshot/.tmp directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7240) Cleanup old snapshots on start
[ https://issues.apache.org/jira/browse/HBASE-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506992#comment-13506992 ] Jesse Yates commented on HBASE-7240: It would be a delete of the /hbase/snapshot/.tmp directory. Might change it we continue running snapshots across master failure sometime in the future, but for the moment, its just a single RPC. Cleanup old snapshots on start -- Key: HBASE-7240 URL: https://issues.apache.org/jira/browse/HBASE-7240 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Fix For: hbase-6055 If the master is hard stopped (i.e. kill -9), the snapshot handler or SnapshotManager may not have a chance to cleanup after the snapshot, leaving extraneous files in the working snapshot directory (/hbase/.snapshot/.tmp directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7242) Use Runtime.exit() instead of Runtime.halt() upon HLog Sync failures
[ https://issues.apache.org/jira/browse/HBASE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506993#comment-13506993 ] Kannan Muthukkaruppan commented on HBASE-7242: -- Currently, don't the shutdown hooks also try to flush/close the regions before closing the ZK connection? Use Runtime.exit() instead of Runtime.halt() upon HLog Sync failures Key: HBASE-7242 URL: https://issues.apache.org/jira/browse/HBASE-7242 Project: HBase Issue Type: Brainstorming Reporter: Amitanand Aiyer Priority: Minor Hey Guys, Should we use Runtime.exit() instead of Runtime.halt(), when we fail a Hlog sync. The key difference is that Runtime.exit() is going to invoke the shutdown hooks; while Runtime.halt() does not. Why we might need this: We had a HDFS name node reboot today on one of our cells, and this caused multiple region servers to abort because they could not sync the Hlog. However, since multiple RS died simultaneously, this seemed like a co-related failure to the master. The master waits for the Znode to expire; but, this could take up to few minutes after RS death (this setting is in place so that we can withstand rack switch reboots, lasting a couple of minutes, without region movement). If the shutdown hooks are called, RS will close the ZK connection, causing a immediate Znode expiry. This might help cut down the unavailability as Regions can begin to get assigned faster. While, we do want to abort on Hlog failure, I do not think it would hurt giving the JVM a few seconds to shutdown gracefully. Please let me know If I am missing something. Thanks, -Amit -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7232: - Attachment: 7232v3.txt Address reviewers' comments and make it actually pass tests. Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt, 7232v3.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7232) Remove HbaseMapWritable
[ https://issues.apache.org/jira/browse/HBASE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7232: - Attachment: 7232v4.txt Was missing files. Remove HbaseMapWritable --- Key: HBASE-7232 URL: https://issues.apache.org/jira/browse/HBASE-7232 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Attachments: 7232.txt, 7232.txt, 7232v2.txt, 7232v3.txt, 7232v4.txt Its used by hfile fileinfo only so need to convert fileinfo to remove this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7202) Family Store Files are not archived on admin.deleteColumn()
[ https://issues.apache.org/jira/browse/HBASE-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7202: - Fix Version/s: (was: 0.94.3) 0.94.4 Family Store Files are not archived on admin.deleteColumn() --- Key: HBASE-7202 URL: https://issues.apache.org/jira/browse/HBASE-7202 Project: HBase Issue Type: Bug Affects Versions: 0.94.2, 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7202-v1.patch, HBASE-7202-v2.patch using HBaseAdmin.deleteColumn() the files are not archived but deleted directory. This causes problems with snapshots, and other systems that relies on files to be archived. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7226) HRegion.checkAndMutate uses incorrect comparison result for , =, and =
[
https://issues.apache.org/jira/browse/HBASE-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7226:
-
Fix Version/s: (was: 0.94.2)
0.94.4
HRegion.checkAndMutate uses incorrect comparison result for , =, and =
---
Key: HBASE-7226
URL: https://issues.apache.org/jira/browse/HBASE-7226
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.2
Environment: 0.94.2
Reporter: Feng Honghua
Priority: Minor
Fix For: 0.94.4
Attachments: HRegion_HBASE_7226_0.94.2.patch
Original Estimate: 10m
Remaining Estimate: 10m
in HRegion.checkAndMutate, incorrect comparison results are used for , =,
and =, as below:
switch (compareOp) {
case LESS:
matches = compareResult = 0; // should be '' here
break;
case LESS_OR_EQUAL:
matches = compareResult 0; // should be '=' here
break;
case EQUAL:
matches = compareResult == 0;
break;
case NOT_EQUAL:
matches = compareResult != 0;
break;
case GREATER_OR_EQUAL:
matches = compareResult 0; // should be '=' here
break;
case GREATER:
matches = compareResult = 0; // should be '' here
break;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507019#comment-13507019 ] Ted Yu commented on HBASE-7236: --- I saw quite some changes which only affect white space. Can you simplify your patch so that it is easier to review ? Thanks add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7236-PROTOTYPE.patch Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Attachment: trunk-6423_v3.2.patch Writes should not block reads on blocking updates to memstores -- Key: HBASE-6423 URL: https://issues.apache.org/jira/browse/HBASE-6423 Project: HBase Issue Type: Bug Reporter: Karthik Ranganathan Assignee: Jimmy Xiang Attachments: trunk-6423.patch, trunk-6423_v2.1.patch, trunk-6423_v2.patch, trunk-6423_v3.2.patch We have a big data use case where we turn off WAL and have a ton of reads and writes. We found that: 1. flushing a memstore takes a while (GZIP compression) 2. incoming writes cause the new memstore to grow in an unbounded fashion 3. this triggers blocking memstore updates 4. in turn, this causes all the RPC handler threads to block on writes to that memstore 5. we are not able to read during this time as RPC handlers are blocked At a higher level, we should not hold up the RPC threads while blocking updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[
https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507028#comment-13507028
]
Lars Hofhansl commented on HBASE-6423:
--
This part's a bit ugly:
{code}
+ protected Configuration createConf() {
+Configuration conf = HBaseConfiguration.create();
+if (busyWaitDuration != null) {
+ conf.set(hbase.busy.wait.duration, busyWaitDuration);
+}
+return conf;
+ }
{code}
We're using the configuration pass this information around? Why isn't it in the
configuration in the first place?
Writes should not block reads on blocking updates to memstores
--
Key: HBASE-6423
URL: https://issues.apache.org/jira/browse/HBASE-6423
Project: HBase
Issue Type: Bug
Reporter: Karthik Ranganathan
Assignee: Jimmy Xiang
Attachments: trunk-6423.patch, trunk-6423_v2.1.patch,
trunk-6423_v2.patch, trunk-6423_v3.2.patch
We have a big data use case where we turn off WAL and have a ton of reads and
writes. We found that:
1. flushing a memstore takes a while (GZIP compression)
2. incoming writes cause the new memstore to grow in an unbounded fashion
3. this triggers blocking memstore updates
4. in turn, this causes all the RPC handler threads to block on writes to
that memstore
5. we are not able to read during this time as RPC handlers are blocked
At a higher level, we should not hold up the RPC threads while blocking
updates, and we should build in some sort of rate control.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
