Changing my vote back to +1, sorry for that. Workload E tests on a new testbed instance with careful attention to configuration do not produce the same results. We have instead:
* Workload E* - 0.98.4RC0 [OVERALL]RunTime(ms) 1270229[OVERALL]Throughput(ops/sec) 7944[INSERT] Operations 499175 [INSERT]AverageLatency(us) 18[INSERT] MinLatency(us) 5 [INSERT]MaxLatency(us) 571160[INSERT]95thPercentileLatency(ms) 0[INSERT] 99thPercentileLatency(ms) 0 [SCAN]Operations 9500825[SCAN] AverageLatency(us) 21089[SCAN] MinLatency(us) 772 [SCAN]MaxLatency(us) 3300020[SCAN]95thPercentileLatency(ms) 107[SCAN] 99thPercentileLatency(ms) 152 I ran workload E a few times to insure the results were consistent. They vary a bit due to natural variance but not by 23%. I don't have yesterday's test cluster around. I strongly suspect I made an unseen error setting up for that workload. On Thu, Jul 17, 2014 at 8:24 PM, Andrew Purtell <[email protected]> wrote: > The difference observed on the remote testbed doesn't show up in > all-localhost testing: > > HEAD > > [SCAN], Operations, 949967 > [SCAN], AverageLatency(us), 23847.456127423375 > [SCAN], MinLatency(us), 625 > [SCAN], MaxLatency(us), 1806981 > [SCAN], 95thPercentileLatency(ms), 56 > [SCAN], 99thPercentileLatency(ms), 71 > > > HEAD~50: 5f853cb... HBASE-11436 > > [SCAN], Operations, 949937 > [SCAN], AverageLatency(us), 23844.437741660764 > [SCAN], MinLatency(us), 961 > [SCAN], MaxLatency(us), 1843125 > [SCAN], 95thPercentileLatency(ms), 55 > [SCAN], 99thPercentileLatency(ms), 70 > > > 0ca0ced Update CHANGES.txt for 0.98.3RC1 > > [SCAN], Operations, 950224 > [SCAN], AverageLatency(us), 24303.889086152318 > [SCAN], MinLatency(us), 956 > [SCAN], MaxLatency(us), 2091141 > [SCAN], 95thPercentileLatency(ms), 56 > [SCAN], 99thPercentileLatency(ms), 71 > > > I have the testbed for one more day. I'll try an educated guess. > Otherwise, will need to change my vote back to +1 because I can't veto a RC > for something I can't verify. > > > > On Thu, Jul 17, 2014 at 5:51 PM, Andrew Purtell <[email protected]> > wrote: > >> Bisecting now. >> >> I plan to find and revert the culprit and any related commits, confirm >> improvement with workload E, push those changes back onto the pile for .5, >> and roll .4 RC1 on or before Monday. Phoenix has a release deadline at the >> end of the month and changes in .4 they need (if not the issue). >> >> >> On Thu, Jul 17, 2014 at 5:12 PM, Andrew Purtell <[email protected]> >> wrote: >> >>> See thread on dev@ titled "Comparing the performance of 0.98.4 RC0 and >>> 0.98.0 using YCSB - 23% perf regression in workload E" >>> >>> -1 on this RC for now, pending reproduction and further analysis on a >>> dev box. >>> >>> [...] >>> >>> These tests were run with no security coprocessors installed, using >>> HFile V2. The workload E results are a concern. *It appears we have a >>> 23% decline in measured scan throughput and an 23% increase in average op >>> time from 27 ms to 35 ms. *This does not correspond to any active >>> security feature (though that could worsen results potentially, untested) >>> so is something changed in core code. Other workloads are not affected so >>> this is something specific to scanning. Perhaps delete tracking. >>> >>> [...] >>> >>> *Workload E* >>> >>> >>> >>> >>> >>> >>> [OVERALL] RunTime(ms)16009102078826 [OVERALL]Throughput(ops/sec) 6308 >>> 4835[INSERT] Operations499131500322 [INSERT]AverageLatency(us) 1417 >>> [INSERT] MinLatency(us)55 [INSERT]MaxLatency(us) 506079564468[INSERT] >>> 95thPercentileLatency(ms)0 0[INSERT]99thPercentileLatency(ms) 00 [SCAN] >>> Operations9500869 9499678[SCAN] AverageLatency(us) >>> >>> >>> 2663634620 [SCAN]MinLatency(us) 746755[SCAN] MaxLatency(us)8067864 >>> 4615914[SCAN]95thPercentileLatency(ms) 117136 [SCAN] >>> 99thPercentileLatency(ms)169 187 >>> >>> >>> >>> >>> >>> On Mon, Jul 14, 2014 at 8:38 PM, Andrew Purtell <[email protected]> >>> wrote: >>> >>>> The 1st HBase 0.98.4 release candidate (RC0) is available for download >>>> at http://people.apache.org/~apurtell/0.98.4RC0/ and Maven artifacts >>>> are also available in the temporary repository >>>> https://repository.apache.org/content/repositories/orgapachehbase-1026/ >>>> >>>> Signed with my code signing key D5365CCD. >>>> >>>> The issues resolved in this release can be found here: >>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12326810 >>>> >>>> >>>> Please try out the candidate and vote +1/-1 by midnight Pacific Time >>>> (00:00 -0800 GMT) on July 21 on whether or not we should release this as >>>> 0.98.4. Three +1 votes from PMC will be required to release. >>>> >>>> >> >> >> -- >> Best regards, >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
