I'm testing a change that keeps the change to CompactionTool but drops the unit test. Will let you know how it goes.
On Wed, Apr 17, 2019 at 10:28 AM Xu Cang <xc...@salesforce.com.invalid> wrote: > I just saw this email, Andrew. Should I re-open HBASE-21959? And revert it > before we understand/fix why it caused the test failure? > Regarding the failing test, do you mean this one "TestBlocksRead"? > Thanks, > > Xu > > On Tue, Apr 16, 2019 at 9:47 PM Andrew Purtell <andrew.purt...@gmail.com> > wrote: > > > I've bisected twice and it lands on this commit: > > > > commit 6bc46bb10920c1c335b784b01d2a326db1a3d587 (HEAD, refs/bisect/bad) > > HBASE-21959 CompactionTool should close the store it uses for > > compacting files, in order to properly archive compacted files. > > > > > hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java > > | 2 ++ > > > > > hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionTool.java > > | 100 > > > > At first glance it's hard to see how this change is relevant, but it does > > introduce a new unit test. > > > > > > On Tue, Apr 16, 2019 at 7:48 PM Andrew Purtell <andrew.purt...@gmail.com > > > > wrote: > > > > > I’ve been able to reproduce it sometimes too and am bisecting. It may > be > > > an interaction between test cases, not a failure per se, but does seem > > have > > > a recent cause, as you pointed out. I’ll be looking at it. > > > > > > Thank you for your kind consideration and for revoking your veto. > > > > > > A coprocessor API fix was just committed to branch-1 so I want to roll > a > > > new RC soon to include it. There is also an issue open to improve the > > > behavior of the UI when the profiler link is clicked but system support > > is > > > not available. > > > > > > > On Apr 16, 2019, at 7:40 PM, Yu Li <car...@gmail.com> wrote: > > > > > > > > After more investigation, the ConnectionRefused exception could be > > > > reproduced with "mvn -Dtest=<case_name> test" after a complete run of > > all > > > > cases through "mvn -PrunAllTests clean test", but cannot by a clean > > > > standalone run (with "mvn *clean* test"). So now I'm more convinced > > it's > > > > some kind of environment chaos caused by parallel execution of test > > > cases, > > > > and not a blocker issue. > > > > > > > > @Andrew It seems to me that kerby jar is not included in our binary > > > > package, so I'm not sure whether a new RC is required by HBASE-22219. > > > > Anyway I'd like to revoke my -1 vote now. Thanks. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > >> On Tue, 16 Apr 2019 at 10:19, Yu Li <car...@gmail.com> wrote: > > > >> > > > >> Sorry for the late response due to job priority. > > > >> > > > >> This ConnectionRefused issue cannot be reproduced on my laptop > (MacOS > > > >> 10.14.4) but could on the linux env. And I've checked and confirmed > it > > > >> could pass with 1.4.7/1.4.9 source package but stably failed with > > 1.5.0, > > > >> performing a git bisect now, will report back later. > > > >> > > > >> Best Regards, > > > >> Yu > > > >> > > > >> > > > >> On Sat, 13 Apr 2019 at 00:38, Andrew Purtell < > > andrew.purt...@gmail.com> > > > >> wrote: > > > >> > > > >>> I also see the occasional ConnectionRefused errors. They don’t > > > reproduce > > > >>> if you run the test standalone. I also only see them on a Linux dev > > > host. > > > >>> That may be enough to find by bisect the commit that introduced > this > > > >>> behavior. Working on it. There is a JIRA filed for this one. Search > > for > > > >>> “TestBlocksRead” and label “branch-1”. > > > >>> > > > >>> Thanks for the investigations. > > > >>> > > > >>>> On Apr 12, 2019, at 6:36 AM, Yu Li <car...@gmail.com> wrote: > > > >>>> > > > >>>> Quick updates: > > > >>>> > > > >>>> W/ patch of HBASE-22219 or say upgrading kerby version to 1.0.1, > the > > > >>>> failures listed above in the 1st part of hbase-server disappeared. > > > >>>> > > > >>>> However, in the 2nd part of hbase-server UT there're still many > > > >>>> ConnectionRefused exceptions (17 errors in total) as shown below, > > > which > > > >>>> could be reproduced easily with -Dtest=xxx command on my > > environments, > > > >>>> still checking the root cause. > > > >>>> > > > >>>> [INFO] Running org.apache.hadoop.hbase.regionserver.TestBlocksRead > > > >>>> [ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time > > > elapsed: > > > >>>> 0.853 s <<< FAILURE! - in > > > >>>> org.apache.hadoop.hbase.regionserver.TestBlocksRead > > > >>>> [ERROR] > > > >>>> > > > >>> > > > > > > testBlocksStoredWhenCachingDisabled(org.apache.hadoop.hbase.regionserver.TestBlocksRead) > > > >>>> Time elapsed: 0.17 s <<< ERROR! > > > >>>> java.net.ConnectException: Call From > > > >>>> z05f06378.sqa.zth.tbsite.net/11.163.183.195 to localhost:35669 > > failed > > > >>> on > > > >>>> connection exception: java.net.ConnectException: Connection > refused; > > > For > > > >>>> more details see: > > > >>>> http://wiki.apache.org/hadoop/ConnectionRefused > > > >>>> at > > > >>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > > > >>>> at > > > >>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > > > >>>> Caused by: java.net.ConnectException: Connection refused > > > >>>> at > > > >>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > > > >>>> at > > > >>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > > > >>>> > > > >>>> Best Regards, > > > >>>> Yu > > > >>>> > > > >>>> > > > >>>>> On Fri, 12 Apr 2019 at 13:11, Yu Li <car...@gmail.com> wrote: > > > >>>>> > > > >>>>> I have no doubt that you've run the tests locally before > > announcing a > > > >>>>> release as you're always a great RM boss. And this shows one > value > > of > > > >>>>> verifying release, that different voter has different > environments. > > > >>>>> > > > >>>>> Now I think the failures may be kerberos related, since I > possibly > > > has > > > >>>>> changed some system configuration when doing Flink testing on > this > > > env > > > >>>>> weeks ago. Located one issue (HBASE-22219) which also observed in > > > >>> 1.4.7, > > > >>>>> will further investigate. > > > >>>>> > > > >>>>> Best Regards, > > > >>>>> Yu > > > >>>>> > > > >>>>> > > > >>>>> On Fri, 12 Apr 2019 at 12:38, Andrew Purtell < > > > andrew.purt...@gmail.com > > > >>>> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> “However it's good to find the issue earlier if there > > > >>>>>> really is any, before release announced.” > > > >>>>>> > > > >>>>>> I run the complete unit test suite before announcing a release > > > >>> candidate. > > > >>>>>> Just to be clear. > > > >>>>>> > > > >>>>>> Totally agree we should get these problems sorted before an > actual > > > >>>>>> release. My policy is to cancel a RC if anyone vetoes for this > > > >>> reason... > > > >>>>>> want as much coverage and varying environments as we can manage. > > > >>>>>> > > > >>>>>> Thank you for your help so far and I hope the failures you see > > > result > > > >>> in > > > >>>>>> analysis and fixes that lead to better test stability. > > > >>>>>> > > > >>>>>>> On Apr 11, 2019, at 9:32 PM, Yu Li <car...@gmail.com> wrote: > > > >>>>>>> > > > >>>>>>> Confirmed in 1.4.7 source the listed out cases passed (all in > the > > > 1st > > > >>>>>> part > > > >>>>>>> of hbase-server so the result comes out quickly.)... Also > > confirmed > > > >>> the > > > >>>>>>> test ran order are the same... > > > >>>>>>> > > > >>>>>>> Will try 1.5.0 again to prevent the environment difference > caused > > > by > > > >>>>>> time. > > > >>>>>>> If 1.5.0 still fails, will start to do the git bisect to locate > > the > > > >>>>>> first > > > >>>>>>> bad commit. > > > >>>>>>> > > > >>>>>>> Was also expecting an easy pass and +1 as always to save time > and > > > >>>>>> efforts, > > > >>>>>>> but obvious no luck. However it's good to find the issue > earlier > > if > > > >>>>>> there > > > >>>>>>> really is any, before release announced. > > > >>>>>>> > > > >>>>>>> Best Regards, > > > >>>>>>> Yu > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>> On Fri, 12 Apr 2019 at 12:16, Yu Li <car...@gmail.com> wrote: > > > >>>>>>>> > > > >>>>>>>> Fine, let's focus on verifying whether it's a real problem > > rather > > > >>> than > > > >>>>>>>> arguing about wording, after all that's not my intention... > > > >>>>>>>> > > > >>>>>>>> As mentioned, I participated in the 1.4.7 release vote[1] and > > > IIRC I > > > >>>>>> was > > > >>>>>>>> using the same env and all tests passed w/o issue, that's > where > > my > > > >>>>>> concern > > > >>>>>>>> lies and the main reason I gave a -1 vote. I'm running against > > > 1.4.7 > > > >>>>>> source > > > >>>>>>>> on the same now and let's see the result. > > > >>>>>>>> > > > >>>>>>>> [1] > > > https://www.mail-archive.com/dev@hbase.apache.org/msg51380.html > > > >>>>>>>> > > > >>>>>>>> Best Regards, > > > >>>>>>>> Yu > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Fri, 12 Apr 2019 at 12:05, Andrew Purtell < > > > >>> andrew.purt...@gmail.com > > > >>>>>>> > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> I believe the test execution order matters. We run some tests > > in > > > >>>>>>>>> parallel. The ordering of tests is determined by readdir() > > > results > > > >>>>>> and this > > > >>>>>>>>> differs from host to host and checkout to checkout. So when > you > > > >>> see a > > > >>>>>>>>> repeatable group of failures, that’s great. And when someone > > else > > > >>>>>> doesn’t > > > >>>>>>>>> see those same tests fail, or they cannot be reproduced when > > > >>> running > > > >>>>>> by > > > >>>>>>>>> themselves, the commonly accepted term of art for this is > > > “flaky”. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>>> On Apr 11, 2019, at 8:52 PM, Yu Li <car...@gmail.com> > wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> Sorry but I'd call it "possible environment related problem" > > or > > > >>> "some > > > >>>>>>>>>> feature may not work well in specific environment", rather > > than > > > a > > > >>>>>> flaky. > > > >>>>>>>>>> > > > >>>>>>>>>> Will check against 1.4.7 released source package before > > opening > > > >>> any > > > >>>>>>>>> JIRA. > > > >>>>>>>>>> > > > >>>>>>>>>> Best Regards, > > > >>>>>>>>>> Yu > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> On Fri, 12 Apr 2019 at 11:37, Andrew Purtell < > > > >>>>>> andrew.purt...@gmail.com> > > > >>>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> And if they pass in my environment , then what should we > call > > > it > > > >>>>>> then. > > > >>>>>>>>> I > > > >>>>>>>>>>> have no doubt you are seeing failures. Therefore can you > > please > > > >>> file > > > >>>>>>>>> JIRAs > > > >>>>>>>>>>> and attach information that can help identify a fix. > Thanks. > > > >>>>>>>>>>> > > > >>>>>>>>>>>> On Apr 11, 2019, at 8:35 PM, Yu Li <car...@gmail.com> > > wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I ran the test suite with the > > > >>> -Dsurefire.rerunFailingTestsCount=2 > > > >>>>>>>>> option > > > >>>>>>>>>>>> and on two different env separately, so it sums up to 6 > > times > > > >>>>>> stable > > > >>>>>>>>>>>> failure for each case, and from my perspective this is not > > > >>> flaky. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> IIRC last time when verifying 1.4.7 on the same env no > such > > > >>> issue > > > >>>>>>>>>>> observed, > > > >>>>>>>>>>>> will double check. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>> Yu > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Fri, 12 Apr 2019 at 00:07, Andrew Purtell < > > > >>>>>>>>> andrew.purt...@gmail.com> > > > >>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> There are two failure cases it looks like. And this looks > > > like > > > >>>>>>>>> flakes. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> The wrong FS assertions are not something I see when I > run > > > >>> these > > > >>>>>>>>> tests > > > >>>>>>>>>>>>> myself. I am not able to investigate something I can’t > > > >>> reproduce. > > > >>>>>>>>> What I > > > >>>>>>>>>>>>> suggest is since you can reproduce do a git bisect to > find > > > the > > > >>>>>> commit > > > >>>>>>>>>>> that > > > >>>>>>>>>>>>> introduced the problem. Then we can revert it. As an > > > >>> alternative > > > >>>>>> we > > > >>>>>>>>> can > > > >>>>>>>>>>>>> open a JIRA, report the problem, temporarily @ignore the > > > test, > > > >>> and > > > >>>>>>>>>>>>> continue. This latter option only should be done if we > are > > > >>> fairly > > > >>>>>>>>>>> confident > > > >>>>>>>>>>>>> it is a test only problem. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> The connect exceptions are interesting. I see these > > sometimes > > > >>> when > > > >>>>>>>>> the > > > >>>>>>>>>>>>> suite is executed, not this particular case, but when the > > > >>> failed > > > >>>>>>>>> test is > > > >>>>>>>>>>>>> executed by itself it always passes. It is possible some > > > >>> change to > > > >>>>>>>>>>> classes > > > >>>>>>>>>>>>> related to the minicluster or startup or shutdown timing > > are > > > >>> the > > > >>>>>>>>> cause, > > > >>>>>>>>>>> but > > > >>>>>>>>>>>>> it is test time flaky behavior. I’m not happy about this > > but > > > it > > > >>>>>>>>> doesn’t > > > >>>>>>>>>>>>> actually fail the release because the failure is never > > > >>> repeatable > > > >>>>>>>>> when > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>> test is run standalone. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> In general it would be great if some attention was paid > to > > > test > > > >>>>>>>>>>>>> cleanliness on branch-1. As RM I’m not in a position to > > > insist > > > >>>>>> that > > > >>>>>>>>>>>>> everything is perfect or there will never be another 1.x > > > >>> release, > > > >>>>>>>>>>> certainly > > > >>>>>>>>>>>>> not from branch-1. So, tests which fail repeatedly block > a > > > >>> release > > > >>>>>>>>> IMHO > > > >>>>>>>>>>> but > > > >>>>>>>>>>>>> flakes do not. > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Apr 10, 2019, at 11:20 PM, Yu Li <car...@gmail.com> > > > wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> -1 > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Observed many UT failures when checking the source > package > > > >>> (tried > > > >>>>>>>>>>>>> multiple > > > >>>>>>>>>>>>>> rounds on two different environments, MacOs and Linux, > got > > > the > > > >>>>>> same > > > >>>>>>>>>>>>>> result), including (but not limited to): > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> TestBulkload: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > shouldBulkLoadSingleFamilyHLog(org.apache.hadoop.hbase.regionserver.TestBulkLoad) > > > >>>>>>>>>>>>>> Time elapsed: 0.083 s <<< ERROR! > > > >>>>>>>>>>>>>> java.lang.IllegalArgumentException: Wrong FS: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > file:/var/folders/t6/vch4nh357f98y1wlq09lbm7h0000gn/T/junit1805329913454564189/junit8020757893576011944/data/default/shouldBulkLoadSingleFamilyHLog/8f4a6b584533de2fd1bf3c398dfaac29, > > > >>>>>>>>>>>>>> expected: hdfs://localhost:55938 > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamiliesAndSpecifiedTableName(TestBulkLoad.java:246) > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamilies(TestBulkLoad.java:256) > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.shouldBulkLoadSingleFamilyHLog(TestBulkLoad.java:150) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> TestStoreFile: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) > > > >>>>>>>>>>>>>> Time elapsed: 0.083 s <<< ERROR! > > > >>>>>>>>>>>>>> java.net.ConnectException: Call From localhost/ > 127.0.0.1 > > to > > > >>>>>>>>>>>>> localhost:55938 > > > >>>>>>>>>>>>>> failed on connection exception: > java.net.ConnectException: > > > >>>>>>>>> Connection > > > >>>>>>>>>>>>>> refused; For more details see: > > > >>>>>>>>>>>>>> http://wiki.apache.org/hadoop/ConnectionRefused > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestStoreFile.writeStoreFile(TestStoreFile.java:1047) > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestStoreFile.testCacheOnWriteEvictOnClose(TestStoreFile.java:908) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> TestHFile: > > > >>>>>>>>>>>>>> > testEmptyHFile(org.apache.hadoop.hbase.io.hfile.TestHFile) > > > >>> Time > > > >>>>>>>>>>> elapsed: > > > >>>>>>>>>>>>>> 0.08 s <<< ERROR! > > > >>>>>>>>>>>>>> java.net.ConnectException: Call From > > > >>>>>>>>>>>>>> z05f06378.sqa.zth.tbsite.net/11.163.183.195 to > > > >>> localhost:35529 > > > >>>>>>>>> failed > > > >>>>>>>>>>> on > > > >>>>>>>>>>>>>> connection exception: java.net.ConnectException: > > Connection > > > >>>>>> refused; > > > >>>>>>>>>>> For > > > >>>>>>>>>>>>>> more details see: > > > >>>>>> http://wiki.apache.org/hadoop/ConnectionRefused > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> org.apache.hadoop.hbase.io > > > >>>>>>>>>>>>> .hfile.TestHFile.testEmptyHFile(TestHFile.java:90) > > > >>>>>>>>>>>>>> Caused by: java.net.ConnectException: Connection refused > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> org.apache.hadoop.hbase.io > > > >>>>>>>>>>>>> .hfile.TestHFile.testEmptyHFile(TestHFile.java:90) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> TestBlocksScanned: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > > > >>>>>>>>>>>>>> Time elapsed: 0.069 s <<< ERROR! > > > >>>>>>>>>>>>>> java.lang.IllegalArgumentException: Wrong FS: > > > >>>>>>>>>>> hdfs://localhost:35529/tmp/ > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > hbase-jueding.ly/hbase/data/default/TestBlocksScannedWithEncoding/a4a416cc3060d9820a621c294af0aa08 > > > >>>>>>>>>>>>> , > > > >>>>>>>>>>>>>> expected: file:/// > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksScanned._testBlocksScanned(TestBlocksScanned.java:90) > > > >>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > org.apache.hadoop.hbase.regionserver.TestBlocksScanned.testBlocksScannedWithEncoding(TestBlocksScanned.java:86) > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> And please let me know if any known issue I'm not aware > > of. > > > >>>>>> Thanks. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>> Yu > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On Mon, 8 Apr 2019 at 11:38, Yu Li <car...@gmail.com> > > > wrote: > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> The performance report LGTM, thanks! (and sorry for the > > lag > > > >>> due > > > >>>>>> to > > > >>>>>>>>>>>>>>> Qingming Festival Holiday here in China) > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Still verifying the release, just some quick feedback: > > > >>> observed > > > >>>>>>>>> some > > > >>>>>>>>>>>>>>> incompatible changes in compatibility report including > > > >>>>>>>>>>>>>>> HBASE-21492/HBASE-21684 and worth a reminder in > > > ReleaseNote. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Irrelative but noticeable: the 1.4.9 release note URL > is > > > >>>>>> invalid on > > > >>>>>>>>>>>>>>> https://hbase.apache.org/downloads.html > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>> Yu > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On Fri, 5 Apr 2019 at 08:45, Andrew Purtell < > > > >>>>>> apurt...@apache.org> > > > >>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> The difference is basically noise per the usual YCSB > > > >>>>>> evaluation. > > > >>>>>>>>>>> Small > > > >>>>>>>>>>>>>>>> differences in workloads D and F (slightly worse) and > > > >>> workload > > > >>>>>> E > > > >>>>>>>>>>>>> (slightly > > > >>>>>>>>>>>>>>>> better) that do not indicate serious regression. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Linux version 4.14.55-62.37.amzn1.x86_64 > > > >>>>>>>>>>>>>>>> c3.8xlarge x 5 > > > >>>>>>>>>>>>>>>> OpenJDK Runtime Environment (build > > > 1.8.0_181-shenandoah-b13) > > > >>>>>>>>>>>>>>>> -Xms20g -Xmx20g -XX:+UseG1GC -XX:+AlwaysPreTouch > > > >>> -XX:+UseNUMA > > > >>>>>>>>>>>>>>>> -XX:-UseBiasedLocking -XX:+ParallelRefProcEnabled > > > >>>>>>>>>>>>>>>> Hadoop 2.9.2 > > > >>>>>>>>>>>>>>>> Init: Load 100 M rows and snapshot > > > >>>>>>>>>>>>>>>> Run: Delete table, clone and redeploy from snapshot, > run > > > 10 > > > >>> M > > > >>>>>>>>>>>>> operations > > > >>>>>>>>>>>>>>>> Args: -threads 100 -target 50000 > > > >>>>>>>>>>>>>>>> Test table: {NAME => 'u', BLOOMFILTER => 'ROW', > VERSIONS > > > => > > > >>>>>> '1', > > > >>>>>>>>>>>>> IN_MEMORY > > > >>>>>>>>>>>>>>>> => 'false', KEEP_DELETED_CELLS => 'FALSE', > > > >>> DATA_BLOCK_ENCODING > > > >>>>>> => > > > >>>>>>>>>>>>>>>> 'ROW_INDEX_V1', TTL => 'FOREVER', COMPRESSION => > > 'SNAPPY', > > > >>>>>>>>>>>>> MIN_VERSIONS => > > > >>>>>>>>>>>>>>>> '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', > > > >>>>>>>>> REPLICATION_SCOPE => > > > >>>>>>>>>>>>>>>> '0'} > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload A > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200592 200583 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49852 49855 > > > >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 544 559 > > > >>>>>>>>>>>>>>>> [READ], MinLatency(us) 267 292 > > > >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 165631 185087 > > > >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 738 742 > > > >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us), 1877 1961 > > > >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1370 1181 > > > >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 702 646 > > > >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 180735 177279 > > > >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 1943 1652 > > > >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 3257 3085 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload B > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200599 200581 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49850 49855 > > > >>>>>>>>>>>>>>>> [READ], AverageLatency(us), 454 471 > > > >>>>>>>>>>>>>>>> [READ], MinLatency(us) 203 213 > > > >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 183423 174207 > > > >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 563 599 > > > >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 1360 1172 > > > >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1064 1029 > > > >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 746 726 > > > >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 163455 101631 > > > >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 1327 1157 > > > >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 2241 1898 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload C > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200541 200538 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49865 49865 > > > >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 332 327 > > > >>>>>>>>>>>>>>>> [READ], MinLatency(us) 175 179 > > > >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 210559 170367 > > > >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 410 396 > > > >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 871 892 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload D > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200579 200562 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49855 49859 > > > >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 487 547 > > > >>>>>>>>>>>>>>>> [READ], MinLatency(us) 210 214 > > > >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 192255 177535 > > > >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 973 1529 > > > >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 1836 2683 > > > >>>>>>>>>>>>>>>> [INSERT], AverageLatency(us) 1239 1152 > > > >>>>>>>>>>>>>>>> [INSERT], MinLatency(us) 807 788 > > > >>>>>>>>>>>>>>>> [INSERT], MaxLatency(us) 184575 148735 > > > >>>>>>>>>>>>>>>> [INSERT], 95thPercentileLatency(us) 1496 1243 > > > >>>>>>>>>>>>>>>> [INSERT], 99thPercentileLatency(us) 2965 2495 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload E > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 10k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 100605 100568 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 9939 9943 > > > >>>>>>>>>>>>>>>> [SCAN], AverageLatency(us) 3548 2687 > > > >>>>>>>>>>>>>>>> [SCAN], MinLatency(us) 696 678 > > > >>>>>>>>>>>>>>>> [SCAN], MaxLatency(us) 1059839 238463 > > > >>>>>>>>>>>>>>>> [SCAN], 95thPercentileLatency(us) 8327 6791 > > > >>>>>>>>>>>>>>>> [SCAN], 99thPercentileLatency(us) 17647 14415 > > > >>>>>>>>>>>>>>>> [INSERT], AverageLatency(us) 2688 1555 > > > >>>>>>>>>>>>>>>> [INSERT], MinLatency(us) 887 815 > > > >>>>>>>>>>>>>>>> [INSERT], MaxLatency(us) 173311 154623 > > > >>>>>>>>>>>>>>>> [INSERT], 95thPercentileLatency(us) 4455 2571 > > > >>>>>>>>>>>>>>>> [INSERT], 99thPercentileLatency(us) 9303 5375 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> YCSB Workload F > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> target 50k/op/s 1.4.9 1.5.0 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> [OVERALL], RunTime(ms) 200562 204178 > > > >>>>>>>>>>>>>>>> [OVERALL], Throughput(ops/sec) 49859 48976 > > > >>>>>>>>>>>>>>>> [READ], AverageLatency(us) 856 1137 > > > >>>>>>>>>>>>>>>> [READ], MinLatency(us) 262 257 > > > >>>>>>>>>>>>>>>> [READ], MaxLatency(us) 205567 222335 > > > >>>>>>>>>>>>>>>> [READ], 95thPercentileLatency(us) 2365 3475 > > > >>>>>>>>>>>>>>>> [READ], 99thPercentileLatency(us) 3099 4143 > > > >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], AverageLatency(us) 2559 2917 > > > >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], MinLatency(us) 1100 1034 > > > >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], MaxLatency(us) 208767 204799 > > > >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], 95thPercentileLatency(us) 5747 > 7627 > > > >>>>>>>>>>>>>>>> [READ-MODIFY-WRITE], 99thPercentileLatency(us) 7203 > 8919 > > > >>>>>>>>>>>>>>>> [UPDATE], AverageLatency(us) 1700 1777 > > > >>>>>>>>>>>>>>>> [UPDATE], MinLatency(us) 737 687 > > > >>>>>>>>>>>>>>>> [UPDATE], MaxLatency(us) 97983 94271 > > > >>>>>>>>>>>>>>>> [UPDATE], 95thPercentileLatency(us) 3377 4147 > > > >>>>>>>>>>>>>>>> [UPDATE], 99thPercentileLatency(us) 4147 4831 > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On Thu, Apr 4, 2019 at 1:14 AM Yu Li < > car...@gmail.com > > > > > > >>>>>> wrote: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Thanks for the efforts boss. > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Since it's a new minor release, do we have > performance > > > >>>>>> comparison > > > >>>>>>>>>>>>> report > > > >>>>>>>>>>>>>>>>> with 1.4.9 as we did when releasing 1.4.0? If so, any > > > >>>>>> reference? > > > >>>>>>>>>>> Many > > > >>>>>>>>>>>>>>>>> thanks! > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Best Regards, > > > >>>>>>>>>>>>>>>>> Yu > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On Thu, 4 Apr 2019 at 07:44, Andrew Purtell < > > > >>>>>> apurt...@apache.org > > > >>>>>>>>>> > > > >>>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> The fourth HBase 1.5.0 release candidate (RC3) is > > > >>> available > > > >>>>>> for > > > >>>>>>>>>>>>>>>> download > > > >>>>>>>>>>>>>>>>> at > > > >>>>>>>>>>>>>>>>>> > > > >>> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/ > > > >>>>>>>>> and > > > >>>>>>>>>>>>>>>> Maven > > > >>>>>>>>>>>>>>>>>> artifacts are available in the temporary repository > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > https://repository.apache.org/content/repositories/orgapachehbase-1292/ > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> The git tag corresponding to the candidate is > > '1.5.0RC3’ > > > >>>>>>>>>>>>> (b0bc7225c5). > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> A detailed source and binary compatibility report > for > > > this > > > >>>>>>>>> release > > > >>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>> available for your review at > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>> > > > >>> > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/compat-check-report.html > > > >>>>>>>>>>>>>>>>>> . > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> A list of the 115 issues resolved in this release > can > > be > > > >>>>>> found > > > >>>>>>>>> at > > > >>>>>>>>>>>>>>>>>> https://s.apache.org/K4Wk . The 1.5.0 changelog is > > > >>> derived > > > >>>>>> from > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>> changelog of the last branch-1.4 release, 1.4.9. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Please try out the candidate and vote +1/0/-1. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> The vote will be open for at least 72 hours. Unless > > > >>>>>> objection I > > > >>>>>>>>>>> will > > > >>>>>>>>>>>>>>>> try > > > >>>>>>>>>>>>>>>>> to > > > >>>>>>>>>>>>>>>>>> close it Friday April 12, 2019 if we have sufficient > > > >>> votes. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Prior to making this announcement I made the > following > > > >>>>>> preflight > > > >>>>>>>>>>>>>>>> checks: > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> RAT check passes (7u80) > > > >>>>>>>>>>>>>>>>>> Unit test suite passes (7u80, 8u181)* > > > >>>>>>>>>>>>>>>>>> Opened the UI in a browser, poked around > > > >>>>>>>>>>>>>>>>>> LTT load 100M rows with 100% verification and 20% > > > updates > > > >>>>>>>>> (8u181) > > > >>>>>>>>>>>>>>>>>> ITBLL 1B rows with slowDeterministic monkey (8u181) > > > >>>>>>>>>>>>>>>>>> ITBLL 1B rows with serverKilling monkey (8u181) > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> There are known flaky tests. See HBASE-21904 and > > > >>> HBASE-21905. > > > >>>>>>>>> These > > > >>>>>>>>>>>>>>>> flaky > > > >>>>>>>>>>>>>>>>>> tests do not represent serious test failures that > > would > > > >>>>>> prevent > > > >>>>>>>>> a > > > >>>>>>>>>>>>>>>>> release. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>>>>>>> Andrew > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> -- > > > >>>>>>>>>>>>>>>> Best regards, > > > >>>>>>>>>>>>>>>> Andrew > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Words like orphans lost among the crosstalk, meaning > > torn > > > >>> from > > > >>>>>>>>>>> truth's > > > >>>>>>>>>>>>>>>> decrepit hands > > > >>>>>>>>>>>>>>>> - A23, Crosstalk > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > >> > > > > > > > > > -- > > Best regards, > > Andrew > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > decrepit hands > > - A23, Crosstalk > > > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
