To make the read comparison fair, I think '--presplit=' should be used. Cheers
On Sun, Feb 23, 2014 at 11:15 AM, lars hofhansl <[email protected]> wrote: > Thanks for the thorough testing. > > > Can you elaborate "page is split according to the MAX_FILESIZE parameter > until all regions are according to it"? > As for the 15% slowdown in the random read test... I have looked through > the issues and did not see anything warranting that, it's also nothing I > had observed in my testing. > > Did you run this with M/R? HBASE-10501 will reduce the default number of > regions per RegionServer (would be as high as 19 per table before even with > little data), so maybe you're seeing a bit less parallelism (for small data > sets). > > > -- Lars > > > > ________________________________ > From: Jean-Marc Spaggiari <[email protected]> > To: dev <[email protected]> > Sent: Sunday, February 23, 2014 5:47 AM > Subject: Re: [VOTE] The 2nd hbase 0.94.17 release candidate is available > for download > > > Haha ;) Yes. UI. Under the page details in the WebUI, In previous release > (maybe not 0.94.16 because I did not test it) when clicking split in the > WebUI, table was split in half. Now, page is split according to the > MAX_FILESIZE > parameter until all regions are according to it. > > I'm not 100% sure about it so might be wrong. But I'm not sure I have seen > that before. > > > > 2014-02-23 8:32 GMT-05:00 Ted Yu <[email protected]>: > > > bq. Click split under UA > > > > I guess you meant 'under UI' > > > > Cheers > > > > On Feb 23, 2014, at 4:47 AM, Jean-Marc Spaggiari < > [email protected]> > > wrote: > > > > > Tested, tl.tr: +1 ;) > > > > > > All tests are done with Oracle JDK 1.7.0_45 + Hadoop 1.2.1 (Except for > > the > > > 0.96.1 test where Hadoop 2.2.0 is used) > > > > > > Checked the jars, the signature, the CHANGES.TXT file, the > documentation > > => > > > Passed. > > > Checked logs, UI, HBCK, test suite => Passed > > > Run all Performances tests => Passed. 0.94.17 performances just a bit > > under > > > 0.94.16. RandomReadTest 15% slower with 0.94.17. Will retest it. Will > > post > > > all the numbers in detail in my blog later today. > > > > > > Only concern is again the IntegrationTestLoadAndVerify result where I > > don't > > > have the exptected result, but it's the same thing with 0.94.16 and > > 0.96.1 > > > too. With difference Hadoop version. > > > > > > > > > 0.94.16 + Hadoop 1.2.1: > > > 14/02/21 21:22:19 INFO mapred.JobClient: > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters > > > 14/02/21 21:22:19 INFO mapred.JobClient: ROWS_WRITTEN=0 > > > 14/02/21 21:22:19 INFO mapred.JobClient: REFERENCES_CHECKED=9855584 > > > > > > 0.94.17 + Hadoop 1.2.1: > > > 14/02/21 21:31:28 INFO mapred.JobClient: > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters > > > 14/02/21 21:31:28 INFO mapred.JobClient: ROWS_WRITTEN=0 > > > 14/02/21 21:31:28 INFO mapred.JobClient: REFERENCES_CHECKED=9855909 > > > > > > 0.96.1 + Hadoop 2.2.0: > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters > > > REFERENCES_CHECKED=9855588 > > > > > > Create table, put, scan, compact => Passed. > > > Offline merged all regions into a single one, hbck, split => Passed. > > > Rolling restart => Failed once because of jar push issue (my bad), > > retried > > > => Passed. > > > Ran all tests (LoadTestTool, IntegrationTestLoadAndVerify, > > > HLogPerformanceEvaluation, IntegrationTestBigLinkedList) on a 4 nodes > > > cluster => Passed. > > > When we click split in the UA, seems to trigger major compaction, which > > was > > > not the case before (Not sure in which version this has been > introduced) > > > > > > Summary: > > > - IntgrationTestLoadAndVerify results still "strange"; > > > - Split behaviour in the UA seem the have changed. > > > - +1 > > > > > > JM > > > PS: Go Canada Go! > > > > > > 2014-02-21 10:01 GMT-05:00 Ted Yu <[email protected]>: > > > > > >> +1 based on the following: > > >> > > >> 1. Ran test suite using JDK 1.7 - pass > > >> 2. compiled Phoenix with locally built 0.94.17 - pass > > >> 3. checked the documentation > > >> > > >> Cheers > > >> > > >> > > >> On Thu, Feb 20, 2014 at 11:18 PM, lars hofhansl <[email protected]> > > wrote: > > >> > > >>> +1 > > >>> > > >>> - checked documentation and tarball > > >>> > > >>> - Ran in local and distributes mode > > >>> - checked the UI pages > > >>> > > >>> - loaded 100m KeyValues > > >>> - tested scan performance > > >>> > > >>> - converted between FAST_DIFF and NONE encoding > > >>> - tested flushes and compactions > > >>> > > >>> > > >>> -- Lars > > >>> > > >>> > > >>> ________________________________ > > >>> From: lars hofhansl <[email protected]> > > >>> To: hbase-dev <[email protected]> > > >>> Sent: Tuesday, February 18, 2014 8:31 PM > > >>> Subject: [VOTE] The 2nd hbase 0.94.17 release candidate is available > > for > > >>> download > > >>> > > >>> > > >>> The 2nd 0.94.17 RC is available for download at > > >>> http://people.apache.org/~larsh/hbase-0.94.17-rc1/ > > >>> Signed with my code signing key: C7CFE328 > > >>> > > >>> HBase 0.94.17 is a bug fix release with 35 fixes: > > >>> [HBASE-7963] - HBase VerifyReplication not working when security > > >>> enabled > > >>> [HBASE-10249] - TestReplicationSyncUpTool fails because failover > > >> takes > > >>> too long > > >>> [HBASE-10274] - MiniZookeeperCluster should close ZKDatabase when > > >>> shutdown ZooKeeperServers > > >>> [HBASE-10319] - HLog should roll periodically to allow DN > > >> decommission > > >>> to eventually complete. > > >>> [HBASE-10320] - Avoid ArrayList.iterator() ExplicitColumnTracker > > >>> [HBASE-10335] - AuthFailedException in zookeeper may block > > >> replication > > >>> forever > > >>> [HBASE-10340] - [BACKPORT] HBASE-9892 Add info port to ServerName > to > > >>> support multi instances in a node > > >>> [HBASE-10363] - [0.94] TestInputSampler and TestInputSamplerTool > > fail > > >>> under hadoop 2.0/23 profiles. > > >>> [HBASE-10371] - Compaction creates empty hfile, then selects this > > >> file > > >>> for compaction and creates empty hfile and over again > > >>> [HBASE-10383] - Secure Bulk Load for 'completebulkload' fails for > > >>> version 0.94.15 > > >>> [HBASE-10400] - [hbck] Continue if region dir missing on region > > merge > > >>> attempt > > >>> [HBASE-10401] - [hbck] perform overlap group merges in parallel > > >>> [HBASE-10448] - ZKUtil create and watch methods don't set watch in > > >>> some cases > > >>> [HBASE-10470] - Import generates huge log file while importing > large > > >>> amounts of data > > >>> [HBASE-10481] - API Compatibility JDiff script does not properly > > >>> handle arguments in reverse order > > >>> [HBASE-10482] - ReplicationSyncUp doesn't clean up its ZK, needed > > for > > >>> tests > > >>> [HBASE-10485] - PrefixFilter#filterKeyValue() should perform > > >> filtering > > >>> on row key > > >>> [HBASE-10489] - TestImportExport fails in 0.94 with Hadoop2 > > >>> [HBASE-10493] - InclusiveStopFilter#filterKeyValue() should > perform > > >>> filtering on row key > > >>> [HBASE-10501] - Improve IncreasingToUpperBoundRegionSplitPolicy to > > >>> avoid too many regions > > >>> [HBASE-10505] - Import.filterKv does not call Filter.filterRowKey > > >>> [HBASE-10506] - Fail-fast if client connection is lost before the > > >> real > > >>> call be executed in RPC layer > > >>> [HBASE-10508] - Backport HBASE-10365 'HBaseFsck should clean up > > >>> connection properly when repair is completed' to 0.94 and 0.96 > > >>> [HBASE-10539] - HRegion.addAndGetGlobalMemstoreSize returns > previous > > >>> size > > >>> [HBASE-10545] - RS Hangs waiting on region to close on shutdown; > has > > >>> to timeout before can go down > > >>> [HBASE-10546] - Two scanner objects are open for each hbase map > task > > >>> but only one scanner object is closed > > >>> [HBASE-10551] - Change local mode back to one RS thread by default > > >>> [HBASE-10552] - > > >> HFilePerformanceEvaluation.GaussianRandomReadBenchmark > > >>> fails sometimes. > > >>> [HBASE-10555] - Backport HBASE-8519 to 0.94, Backup master will > > never > > >>> come up if primary master dies during initialization > > >>> [HBASE-10562] - Fix TestMultiTableInputFormat for Hadoop 2 in 0.94 > > >>> [HBASE-10212] - New rpc metric: number of active handler > > >>> [HBASE-10423] - Report back the message of split or rollback > failure > > >>> to the master > > >>> [HBASE-10457] - Print corrupted file information in SnapshotInfo > > tool > > >>> without -file option > > >>> [HBASE-10473] - Add utility for adorning http Context > > >>> [HBASE-10480] - TestLogRollPeriod#testWithEdits may fail due to > > >>> insufficient waiting > > >>> > > >>> Notable in this release is HBASE-10501, which changes how regions are > > >>> distributed for an initially empty table. > > >>> > > >>> The list of changes is also available here: > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12325845 > > >>> > > >>> Here're the jenkins runs for this RC: > > >>> https://builds.apache.org/job/HBase-0.94.17/15/ and > > >>> https://builds.apache.org/job/HBase-0.94.17-security/5/ > > >>> > > >>> Please try out the RC, check out the doc, take it for a spin, etc, > and > > >>> vote +1/-1 by EOD February 24th on whether we should release this as > > >>> 0.94.17. > > >>> > > >>> As before, I realize that release testing is like pulling teeth, but > > >>> please give this is quick run, so we can get these bug fixes out > there. > > >>> > > >>> Thanks. > > >>> > > >>> -- Lars > > >> > > >
