I ran the test suite with the -Dsurefire.rerunFailingTestsCount=2 option and on two different env separately, so it sums up to 6 times stable failure for each case, and from my perspective this is not flaky.
IIRC last time when verifying 1.4.7 on the same env no such issue observed, will double check. Best Regards, Yu On Fri, 12 Apr 2019 at 00:07, Andrew Purtell <andrew.purt...@gmail.com> wrote: > There are two failure cases it looks like. And this looks like flakes. > > The wrong FS assertions are not something I see when I run these tests > myself. I am not able to investigate something I can’t reproduce. What I > suggest is since you can reproduce do a git bisect to find the commit that > introduced the problem. Then we can revert it. As an alternative we can > open a JIRA, report the problem, temporarily @ignore the test, and > continue. This latter option only should be done if we are fairly confident > it is a test only problem. > > The connect exceptions are interesting. I see these sometimes when the > suite is executed, not this particular case, but when the failed test is > executed by itself it always passes. It is possible some change to classes > related to the minicluster or startup or shutdown timing are the cause, but > it is test time flaky behavior. I’m not happy about this but it doesn’t > actually fail the release because the failure is never repeatable when the > test is run standalone. > > In general it would be great if some attention was paid to test > cleanliness on branch-1. As RM I’m not in a position to insist that > everything is perfect or there will never be another 1.x release, certainly > not from branch-1. So, tests which fail repeatedly block a release IMHO but > flakes do not. > > > > On Apr 10, 2019, at 11:20 PM, Yu Li <car...@gmail.com> wrote: > > > > -1 > > > > Observed many UT failures when checking the source package (tried > multiple > > rounds on two different environments, MacOs and Linux, got the same > > result), including (but not limited to): > > > > TestBulkload: > > > shouldBulkLoadSingleFamilyHLog(org.apache.hadoop.hbase.regionserver.TestBulkLoad) > > Time elapsed: 0.083 s <<< ERROR! > > java.lang.IllegalArgumentException: Wrong FS: > > > file:/var/folders/t6/vch4nh357f98y1wlq09lbm7h0000gn/T/junit1805329913454564189/junit8020757893576011944/data/default/shouldBulkLoadSingleFamilyHLog/8f4a6b584533de2fd1bf3c398dfaac29, > > expected: hdfs://localhost:55938 > > at > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamiliesAndSpecifiedTableName(TestBulkLoad.java:246) > > at > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamilies(TestBulkLoad.java:256) > > at > > > org.apache.hadoop.hbase.regionserver.TestBulkLoad.shouldBulkLoadSingleFamilyHLog(TestBulkLoad.java:150) > > > > TestStoreFile: > > > testCacheOnWriteEvictOnClose(org.apache.hadoop.hbase.regionserver.TestStoreFile) > > Time elapsed: 0.083 s <<< ERROR! > > java.net.ConnectException: Call From localhost/127.0.0.1 to > localhost:55938 > > failed on connection exception: java.net.ConnectException: Connection > > refused; For more details see: > > http://wiki.apache.org/hadoop/ConnectionRefused > > at > > > org.apache.hadoop.hbase.regionserver.TestStoreFile.writeStoreFile(TestStoreFile.java:1047) > > at > > > org.apache.hadoop.hbase.regionserver.TestStoreFile.testCacheOnWriteEvictOnClose(TestStoreFile.java:908) > > > > TestHFile: > > testEmptyHFile(org.apache.hadoop.hbase.io.hfile.TestHFile) Time elapsed: > > 0.08 s <<< ERROR! > > java.net.ConnectException: Call From > > z05f06378.sqa.zth.tbsite.net/11.163.183.195 to localhost:35529 failed on > > connection exception: java.net.ConnectException: Connection refused; For > > more details see: http://wiki.apache.org/hadoop/ConnectionRefused > > at > > org.apache.hadoop.hbase.io > .hfile.TestHFile.testEmptyHFile(TestHFile.java:90) > > Caused by: java.net.ConnectException: Connection refused > > at > > org.apache.hadoop.hbase.io > .hfile.TestHFile.testEmptyHFile(TestHFile.java:90) > > > > TestBlocksScanned: > > > testBlocksScannedWithEncoding(org.apache.hadoop.hbase.regionserver.TestBlocksScanned) > > Time elapsed: 0.069 s <<< ERROR! > > java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:35529/tmp/ > > > hbase-jueding.ly/hbase/data/default/TestBlocksScannedWithEncoding/a4a416cc3060d9820a621c294af0aa08 > , > > expected: file:/// > > at > > > org.apache.hadoop.hbase.regionserver.TestBlocksScanned._testBlocksScanned(TestBlocksScanned.java:90) > > at > > > org.apache.hadoop.hbase.regionserver.TestBlocksScanned.testBlocksScannedWithEncoding(TestBlocksScanned.java:86) > > > > And please let me know if any known issue I'm not aware of. Thanks. > > > > Best Regards, > > Yu > > > > > >> On Mon, 8 Apr 2019 at 11:38, Yu Li <car...@gmail.com> wrote: > >> > >> The performance report LGTM, thanks! (and sorry for the lag due to > >> Qingming Festival Holiday here in China) > >> > >> Still verifying the release, just some quick feedback: observed some > >> incompatible changes in compatibility report including > >> HBASE-21492/HBASE-21684 and worth a reminder in ReleaseNote. > >> > >> Irrelative but noticeable: the 1.4.9 release note URL is invalid on > >> https://hbase.apache.org/downloads.html > >> > >> Best Regards, > >> Yu > >> > >> > >>> On Fri, 5 Apr 2019 at 08:45, Andrew Purtell <apurt...@apache.org> > wrote: > >>> > >>> The difference is basically noise per the usual YCSB evaluation. Small > >>> differences in workloads D and F (slightly worse) and workload E > (slightly > >>> better) that do not indicate serious regression. > >>> > >>> Linux version 4.14.55-62.37.amzn1.x86_64 > >>> c3.8xlarge x 5 > >>> OpenJDK Runtime Environment (build 1.8.0_181-shenandoah-b13) > >>> -Xms20g -Xmx20g -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseNUMA > >>> -XX:-UseBiasedLocking -XX:+ParallelRefProcEnabled > >>> Hadoop 2.9.2 > >>> Init: Load 100 M rows and snapshot > >>> Run: Delete table, clone and redeploy from snapshot, run 10 M > operations > >>> Args: -threads 100 -target 50000 > >>> Test table: {NAME => 'u', BLOOMFILTER => 'ROW', VERSIONS => '1', > IN_MEMORY > >>> => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => > >>> 'ROW_INDEX_V1', TTL => 'FOREVER', COMPRESSION => 'SNAPPY', > MIN_VERSIONS => > >>> '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => > >>> '0'} > >>> > >>> > >>> YCSB Workload A > >>> > >>> target 50k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 200592 200583 > >>> [OVERALL], Throughput(ops/sec) 49852 49855 > >>> [READ], AverageLatency(us) 544 559 > >>> [READ], MinLatency(us) 267 292 > >>> [READ], MaxLatency(us) 165631 185087 > >>> [READ], 95thPercentileLatency(us) 738 742 > >>> [READ], 99thPercentileLatency(us), 1877 1961 > >>> [UPDATE], AverageLatency(us) 1370 1181 > >>> [UPDATE], MinLatency(us) 702 646 > >>> [UPDATE], MaxLatency(us) 180735 177279 > >>> [UPDATE], 95thPercentileLatency(us) 1943 1652 > >>> [UPDATE], 99thPercentileLatency(us) 3257 3085 > >>> > >>> YCSB Workload B > >>> > >>> target 50k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 200599 200581 > >>> [OVERALL], Throughput(ops/sec) 49850 49855 > >>> [READ], AverageLatency(us), 454 471 > >>> [READ], MinLatency(us) 203 213 > >>> [READ], MaxLatency(us) 183423 174207 > >>> [READ], 95thPercentileLatency(us) 563 599 > >>> [READ], 99thPercentileLatency(us) 1360 1172 > >>> [UPDATE], AverageLatency(us) 1064 1029 > >>> [UPDATE], MinLatency(us) 746 726 > >>> [UPDATE], MaxLatency(us) 163455 101631 > >>> [UPDATE], 95thPercentileLatency(us) 1327 1157 > >>> [UPDATE], 99thPercentileLatency(us) 2241 1898 > >>> > >>> YCSB Workload C > >>> > >>> target 50k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 200541 200538 > >>> [OVERALL], Throughput(ops/sec) 49865 49865 > >>> [READ], AverageLatency(us) 332 327 > >>> [READ], MinLatency(us) 175 179 > >>> [READ], MaxLatency(us) 210559 170367 > >>> [READ], 95thPercentileLatency(us) 410 396 > >>> [READ], 99thPercentileLatency(us) 871 892 > >>> > >>> YCSB Workload D > >>> > >>> target 50k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 200579 200562 > >>> [OVERALL], Throughput(ops/sec) 49855 49859 > >>> [READ], AverageLatency(us) 487 547 > >>> [READ], MinLatency(us) 210 214 > >>> [READ], MaxLatency(us) 192255 177535 > >>> [READ], 95thPercentileLatency(us) 973 1529 > >>> [READ], 99thPercentileLatency(us) 1836 2683 > >>> [INSERT], AverageLatency(us) 1239 1152 > >>> [INSERT], MinLatency(us) 807 788 > >>> [INSERT], MaxLatency(us) 184575 148735 > >>> [INSERT], 95thPercentileLatency(us) 1496 1243 > >>> [INSERT], 99thPercentileLatency(us) 2965 2495 > >>> > >>> YCSB Workload E > >>> > >>> target 10k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 100605 100568 > >>> [OVERALL], Throughput(ops/sec) 9939 9943 > >>> [SCAN], AverageLatency(us) 3548 2687 > >>> [SCAN], MinLatency(us) 696 678 > >>> [SCAN], MaxLatency(us) 1059839 238463 > >>> [SCAN], 95thPercentileLatency(us) 8327 6791 > >>> [SCAN], 99thPercentileLatency(us) 17647 14415 > >>> [INSERT], AverageLatency(us) 2688 1555 > >>> [INSERT], MinLatency(us) 887 815 > >>> [INSERT], MaxLatency(us) 173311 154623 > >>> [INSERT], 95thPercentileLatency(us) 4455 2571 > >>> [INSERT], 99thPercentileLatency(us) 9303 5375 > >>> > >>> YCSB Workload F > >>> > >>> target 50k/op/s 1.4.9 1.5.0 > >>> > >>> > >>> > >>> [OVERALL], RunTime(ms) 200562 204178 > >>> [OVERALL], Throughput(ops/sec) 49859 48976 > >>> [READ], AverageLatency(us) 856 1137 > >>> [READ], MinLatency(us) 262 257 > >>> [READ], MaxLatency(us) 205567 222335 > >>> [READ], 95thPercentileLatency(us) 2365 3475 > >>> [READ], 99thPercentileLatency(us) 3099 4143 > >>> [READ-MODIFY-WRITE], AverageLatency(us) 2559 2917 > >>> [READ-MODIFY-WRITE], MinLatency(us) 1100 1034 > >>> [READ-MODIFY-WRITE], MaxLatency(us) 208767 204799 > >>> [READ-MODIFY-WRITE], 95thPercentileLatency(us) 5747 7627 > >>> [READ-MODIFY-WRITE], 99thPercentileLatency(us) 7203 8919 > >>> [UPDATE], AverageLatency(us) 1700 1777 > >>> [UPDATE], MinLatency(us) 737 687 > >>> [UPDATE], MaxLatency(us) 97983 94271 > >>> [UPDATE], 95thPercentileLatency(us) 3377 4147 > >>> [UPDATE], 99thPercentileLatency(us) 4147 4831 > >>> > >>> > >>>> On Thu, Apr 4, 2019 at 1:14 AM Yu Li <car...@gmail.com> wrote: > >>>> > >>>> Thanks for the efforts boss. > >>>> > >>>> Since it's a new minor release, do we have performance comparison > report > >>>> with 1.4.9 as we did when releasing 1.4.0? If so, any reference? Many > >>>> thanks! > >>>> > >>>> Best Regards, > >>>> Yu > >>>> > >>>> > >>>> On Thu, 4 Apr 2019 at 07:44, Andrew Purtell <apurt...@apache.org> > >>> wrote: > >>>> > >>>>> The fourth HBase 1.5.0 release candidate (RC3) is available for > >>> download > >>>> at > >>>>> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/ and > >>> Maven > >>>>> artifacts are available in the temporary repository > >>>>> > >>> > https://repository.apache.org/content/repositories/orgapachehbase-1292/ > >>>>> > >>>>> The git tag corresponding to the candidate is '1.5.0RC3’ > (b0bc7225c5). > >>>>> > >>>>> A detailed source and binary compatibility report for this release is > >>>>> available for your review at > >>>>> > >>>>> > >>>> > >>> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC3/compat-check-report.html > >>>>> . > >>>>> > >>>>> A list of the 115 issues resolved in this release can be found at > >>>>> https://s.apache.org/K4Wk . The 1.5.0 changelog is derived from the > >>>>> changelog of the last branch-1.4 release, 1.4.9. > >>>>> > >>>>> Please try out the candidate and vote +1/0/-1. > >>>>> > >>>>> The vote will be open for at least 72 hours. Unless objection I will > >>> try > >>>> to > >>>>> close it Friday April 12, 2019 if we have sufficient votes. > >>>>> > >>>>> Prior to making this announcement I made the following preflight > >>> checks: > >>>>> > >>>>> RAT check passes (7u80) > >>>>> Unit test suite passes (7u80, 8u181)* > >>>>> Opened the UI in a browser, poked around > >>>>> LTT load 100M rows with 100% verification and 20% updates (8u181) > >>>>> ITBLL 1B rows with slowDeterministic monkey (8u181) > >>>>> ITBLL 1B rows with serverKilling monkey (8u181) > >>>>> > >>>>> There are known flaky tests. See HBASE-21904 and HBASE-21905. These > >>> flaky > >>>>> tests do not represent serious test failures that would prevent a > >>>> release. > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> Andrew > >>>>> > >>>> > >>> > >>> > >>> -- > >>> Best regards, > >>> Andrew > >>> > >>> Words like orphans lost among the crosstalk, meaning torn from truth's > >>> decrepit hands > >>> - A23, Crosstalk > >>> > >> >
