The single node cluster was just a local install, right? I.e. using the local
file system, rather than HDFS...?
On the 7 node cluster, which version of HDFS did you use? If not 1.0.4 I assume
you recompiled HBase :)
I definitely do not see the AbstractMethodError issue. That very looks like a
classpath setup issue.
Ran IntegrationTestLoadAndVerify and IntegrationTestBigLinkedList in a loop in
local mode. Didn't fail once.
Let's chat offline and figure out if/where your setup is different from mine.
-- Lars
________________________________
From: lars hofhansl <la...@apache.org>
To: Jean-Marc Spaggiari <jean-m...@spaggiari.org>; dev <dev@hbase.apache.org>
Sent: Thursday, December 19, 2013 8:53 AM
Subject: Re: [VOTE] The 1st hbase 0.94.15 release candidate is available for
download
Thanks JM.
You did a "raw" scan below. It'll return to you exactly what is there, so
you'll see the 3 versions before you compact, that is by design.
java.lang.AbstractMethodError looks like an issue local to your install. I'll
check.
IntegrationTestLoadAndVerify is interesting. Did that pass reliably in older
releases of 0.94 (0.94.14 or 0.94.13)?
-- Lars
________________________________
From: Jean-Marc Spaggiari <jean-m...@spaggiari.org>
To: dev <dev@hbase.apache.org>; lars hofhansl <la...@apache.org>
Sent: Thursday, December 19, 2013 7:01 AM
Subject: Re: [VOTE] The 1st hbase 0.94.15 release candidate is available for
download
tl;tr see arrow below.
Downloaded and checked signature for bother vanilla and secured. Passed.
Random checked documentation and CHANGES.txt. Passed
On a single node cluster:
Ran the tests. All passed.
Ran IntegrationTestLoadAndVerify. Got REFERENCES_CHECKED=9855424, expected
10000000? Failed?
Ran IntegrationTestBigLinkedList. Passed.
Ran HBCK after those tests and got many errors about _original-evil-name and
clone tables.
Cleared everything, restarted HBase. Re-ran IntegrationTestBigLinkedList, HBCK
ok. Re-ran IntegrationTestLoadAndVerify, failed again:
13/12/18 21:24:24 ERROR test.IntegrationTestBigLinkedList$Verify: Expected
referenced count does not match with actual referenced count. expected
referenced=3000000 ,actual=9000000
Exception in thread "main" java.lang.RuntimeException: Verify.verify failed
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.runVerify(IntegrationTestBigLinkedList.java:724)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Loop.run(IntegrationTestBigLinkedList.java:757)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.run(IntegrationTestBigLinkedList.java:1069)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1073)
But now HBCK is clean. Figured that HBCK issue is because of some leftover from
org.apache.hadoop.hbase.regionserver.TestStoreFile who is writting in the same
directory as the default standalone HBase.
From the shell, create a table 15 regions, put, compact, scan, etc. Table
definition is VERSIONS => 2. However, scan 't1', {RAW => true, VERSIONS => 10}
still return 3 versions even after flush/compact/major_compact:
hbase(main):034:0> scan 't1', {RAW => true, VERSIONS => 10}
ROW COLUMN+CELL
rowkey column=f1:c1,
timestamp=1387421969489, value=value
rowkey column=f1:c1,
timestamp=1387421969337, value=value
rowkey column=f1:c1,
timestamp=1387421969162, value=value
1 row(s) in 0.0570 seconds
Will have expected only 2 to be return.
Stopped HBase, checked the log, everything is fine.
Now on a 7 nodes cluster:
Deployed jars and did rolling restart on a 0.94.14 cluster. Passed.
Configured default balancer, merged a 60 region table to a single region,
restarted the cluster, all fine.
major_compact the table to get it split into 60 regions, balancer, all fine
except that balancer need to be run twice to get correct balancing.
Some "No serialized HRegionInfo in keyvalues" in the logs not related to the
tables I'm "playing" with.
Restored customized balancer, restarted, rebalanced, all fine.
Ran IntegrationTestLoadAndVerify. Got REFERENCES_CHECKED=9855645, expected
10000000? Failed?
Ran IntegrationTestBigLinkedList. Passed.
Last, I tried to run IntegrationTestsDriver but it failed. I need to look at
that.
hbase@node3:~/hbase-0.94.3$ bin/hbase
org.apache.hadoop.hbase.IntegrationTestsDriver
Exception in thread "main" java.lang.AbstractMethodError:
org.apache.hadoop.hbase.util.AbstractHBaseTool.doWork()V
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at
org.apache.hadoop.hbase.IntegrationTestsDriver.main(IntegrationTestsDriver.java:47)
=====> tl;tr:
- Small issue with balancer when 60 regions assigned to a single server. Need
to run twice to get that correctly balanced;
- Leftover in the wrong place from
org.apache.hadoop.hbase.regionserver.TestStoreFile;
- Table with VERSIONS => 2 returns 3 versions instead of 2;
- IntegrationTestsDriver not running.
I don't think there is anything here to stop the release but there is still few
things that need to be looked at.
JM
2013/12/18 lars hofhansl <la...@apache.org>
The 1st 0.94.15 RC is available for download at
http://people.apache.org/~larsh/hbase-0.94.15-rc0/
>Signed with my code signing key: C7CFE328
>
>HBase 0.94.15 is a bug fix release along with some performance improvements:
> [HBASE-7886] - [replication] hlog zk node will not be deleted if client
>roll hlog
> [HBASE-9485] - TableOutputCommitter should implement recovery if we don't
>want jobs to start from 0 on RM restart
> [HBASE-9995] - Not stopping ReplicationSink when using custom
>implementation for the ReplicationSink
> [HBASE-10014] - HRegion#doMiniBatchMutation rollbacks the memstore even if
>there is nothing to rollback.
> [HBASE-10015] - Replace intrinsic locking with explicit locks in
>StoreScanner
> [HBASE-10026] - HBaseAdmin#createTable could fail if region splits too fast
> [HBASE-10046] - Unmonitored HBase service could accumulate Status objects
>and OOM
> [HBASE-10057] - TestRestoreFlushSnapshotFromClient and
>TestRestoreSnapshotFromClient fail to finish occasionally
> [HBASE-10061] - TableMapReduceUtil.findOrCreateJar calls updateMap(null, )
>resulting in thrown NPE
> [HBASE-10064] - AggregateClient.validateParameters can throw NPE
> [HBASE-10089] - Metrics intern table names cause eventual permgen OOM in
>0.94
> [HBASE-10111] - Verify that a snapshot is not corrupted before restoring it
> [HBASE-10112] - Hbase rest query params for maxVersions and maxValues are
>not parsed
> [HBASE-10117] - Avoid synchronization in HRegionScannerImpl.isFilterDone
> [HBASE-10120] - start-hbase.sh doesn't respect --config in non-distributed
>mode
> [HBASE-10179] - HRegionServer underreports readRequestCounts by 1 under
>certain conditions
> [HBASE-10181] - HBaseObjectWritable.readObject catches
>DoNotRetryIOException and wraps it back in a regular IOException
> [HBASE-9931] - Optional setBatch for CopyTable to copy large rows in
>batches
> [HBASE-10001] - Add a coprocessor to help testing the performances without
>taking into account the i/o
> [HBASE-10007] - PerformanceEvaluation: Add sampling and latency collection
>to randomRead test
> [HBASE-10010] - eliminate the put latency spike on the new log file
>beginning
> [HBASE-10048] - Add hlog number metric in regionserver
> [HBASE-10049] - Small improvments in region_mover.rb
> [HBASE-10093] - Unregister ReplicationSource metric bean when the
>replication source thread is terminated
> [HBASE-9047] - Tool to handle finishing replication when the cluster is
>offline
> [HBASE-10119] - Allow HBase coprocessors to clean up when they fail
> [HBASE-9927] - ReplicationLogCleaner#stop() calls
>HConnectionManager#deleteConnection() unnecessarily
> [HBASE-9986] - Incorporate HTTPS support for HBase (0.94 port)
> [HBASE-10058] - Test for HBASE-9915 (avoid reading index blocks)
> [HBASE-10189] - Intermittent TestReplicationSyncUpTool failure
>
>The list of changes is also available here:
>https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12325559
>
>Here're the jenkins runs for this RC:
>https://builds.apache.org/job/HBase-0.94.15/2/ and
>https://builds.apache.org/job/HBase-0.94.15-security/1/
>
>Please try out the RC, check out the doc, take it for a spin, etc, and vote
>+1/-1 by EOD December 27th on whether we should release this as 0.94.15. (9
>days because of the holidays)
>
>Thanks.
>
>-- Lars
>
