updated the precommit patch testing job to remove the failed-build-specific description pattern (it was set to match HDFS jira ids) so that everything will use the hbase jira matching regex.
On Mon, Jun 22, 2015 at 11:48 PM, Sean Busbey <bus...@cloudera.com> wrote: > I moved TestBulkIngest and TestIngest out to HBASE-13952 and HBASE-13953, > respectively. > > added versions of the remaining tests that runs on branch-1 and master. > presuming things come back passing I'll turn on notifications tomorrow. > > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <bus...@cloudera.com> wrote: > >> I've add a new build test to run the ITs under sunny day conditions using >> minicluster for both java 7 and java 8 on the 1.2 line. >> >> https://builds.apache.org/job/HBase-1.2-IT/ >> >> I haven't turned on notifications yet, because BulkIngest and TestIngest >> are read. details on HBASE-13750 >> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <bus...@cloudera.com> wrote: >> >>> sigh. that should have been "is now a matrix build". >>> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <bus...@cloudera.com> >>> wrote: >>> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8 in >>>> parallel. >>>> >>>> I saved the old job for now and left it disabled[2]. >>>> >>>> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/ >>>> [2]: >>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/ >>>> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <bus...@cloudera.com> >>>> wrote: >>>> >>>>> Sorry for the noise. I also updated the checkstyle step on HBase-TRUNK >>>>> >>>>> from >>>>> mvn checkstyle:checkstyle >>>>> to >>>>> mvn -DskipTests package checkstyle:checkstyle >>>>> >>>>> to deal with the same issue. Looks all clear now. >>>>> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <bus...@cloudera.com> >>>>> wrote: >>>>> >>>>>> Updated the following builds: >>>>>> >>>>>> * HBase-TRUNK >>>>>> >>>>>> moved from >>>>>> >>>>>> mvn clean compile findbugs:findbugs >>>>>> >>>>>> to >>>>>> >>>>>> mvn clean -DskipTests package findbugs:findbugs >>>>>> >>>>>> To work around the known issue where we can't do a bootstrap compile >>>>>> without previous install or remote SNAPSHOT artifacts. (recently >>>>>> triggered >>>>>> by the addition of the procedure module on master) >>>>>> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino...@gmail.com> wrote: >>>>>> >>>>>>> Make findbugs and checkstyle plugins always run. >>>>>>> The default behavior only publishes result for stable builds, but >>>>>>> master is >>>>>>> not always stable and sometimes we introduce new warnings in unstable >>>>>>> builds(see builds 6310-6314). >>>>>>> And findbugs and checkstyle will not fail unless we abort the >>>>>>> building >>>>>>> process I think, so it is safe to turn it on every time. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: >>>>>>> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report. >>>>>>> > >>>>>>> > Cheers >>>>>>> > >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino...@gmail.com> wrote: >>>>>>> > >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show >>>>>>> findbugs, >>>>>>> > > checkstyle and jacoco coverage report. >>>>>>> > > The config has been tested on >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly 30 >>>>>>> times. >>>>>>> > Not >>>>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(the >>>>>>> overhead >>>>>>> > of >>>>>>> > > collecting information for code coverage). >>>>>>> > > Thanks. >>>>>>> > > >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurt...@apache.org>: >>>>>>> > > >>>>>>> > > > +1, thanks a lot for improving our build hygiene. >>>>>>> > > > >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar < >>>>>>> enis....@gmail.com> >>>>>>> > > wrote: >>>>>>> > > > >>>>>>> > > > > Thanks Sean. This is great. >>>>>>> > > > > >>>>>>> > > > > Enis >>>>>>> > > > > >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey < >>>>>>> bus...@cloudera.com> >>>>>>> > > > wrote: >>>>>>> > > > > >>>>>>> > > > > > FYI I just finished chasing down the breakage for mvn site >>>>>>> on all >>>>>>> > > patch >>>>>>> > > > > > builds. >>>>>>> > > > > > >>>>>>> > > > > > HBASE-13191 consolidates the few places in the test-patch >>>>>>> code >>>>>>> > where >>>>>>> > > we >>>>>>> > > > > > hard coded MAVEN_OPTS. >>>>>>> > > > > > >>>>>>> > > > > > If you look at the PreCommit job now, we use the "set >>>>>>> environment >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then everything >>>>>>> else >>>>>>> > respects >>>>>>> > > > > that >>>>>>> > > > > > setting. >>>>>>> > > > > > >>>>>>> > > > > > I set the initial value to be a combination of the memory >>>>>>> > limitations >>>>>>> > > > > we've >>>>>>> > > > > > been actually running with (the ~6G was getting ignored) >>>>>>> and the >>>>>>> > > > permgen >>>>>>> > > > > > needed for site. >>>>>>> > > > > > >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData -XX:MaxPermSize=256m >>>>>>> > > > > > >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack <st...@duboce.net> >>>>>>> wrote: >>>>>>> > > > > > >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm as >>>>>>> patch-build >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS to >>>>>>> be the >>>>>>> > > same >>>>>>> > > > as >>>>>>> > > > > > > those of trunk build too, setting >>>>>>> MAVEN_OPTS="-Xmx6100m"... it >>>>>>> > had >>>>>>> > > > been >>>>>>> > > > > > > 3000. >>>>>>> > > > > > > >>>>>>> > > > > > > Yours, >>>>>>> > > > > > > St.Ack >>>>>>> > > > > > > >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack <st...@duboce.net> >>>>>>> wrote: >>>>>>> > > > > > > >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds and >>>>>>> or last >>>>>>> > 7 >>>>>>> > > > > days, >>>>>>> > > > > > > > whichever comes first. >>>>>>> > > > > > > > St.Ack >>>>>>> > > > > > > > >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack < >>>>>>> st...@duboce.net> >>>>>>> > wrote: >>>>>>> > > > > > > > >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run >>>>>>> mostly blue >>>>>>> > > (give >>>>>>> > > > or >>>>>>> > > > > > > take >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue >>>>>>> branch-1 has >>>>>>> > > helped >>>>>>> > > > us >>>>>>> > > > > > > >> identify at least one destabilizing commit in the >>>>>>> last few >>>>>>> > days, >>>>>>> > > > > maybe >>>>>>> > > > > > > two; >>>>>>> > > > > > > >> this is as it should be (smile). >>>>>>> > > > > > > >> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch, >>>>>>> make sure >>>>>>> > > > > subsequent >>>>>>> > > > > > > >> builds stay blue. You can subscribe to >>>>>>> > bui...@hbase.apache.org >>>>>>> > > to >>>>>>> > > > > get >>>>>>> > > > > > > >> notice of failures if not already subscribed. >>>>>>> > > > > > > >> >>>>>>> > > > > > > >> Thanks, >>>>>>> > > > > > > >> St.Ack >>>>>>> > > > > > > >> >>>>>>> > > > > > > >> 1. >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/ >>>>>>> > > > > > > >> 2. >>>>>>> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ >>>>>>> > > > > > > >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack < >>>>>>> st...@duboce.net> >>>>>>> > > wrote: >>>>>>> > > > > > > >> >>>>>>> > > > > > > >>> A few notes on testing. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now and >>>>>>> after some >>>>>>> > > work, >>>>>>> > > > we >>>>>>> > > > > > are >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. Lets >>>>>>> try and >>>>>>> > > keep >>>>>>> > > > it >>>>>>> > > > > > > this >>>>>>> > > > > > > >>> way going forward. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with more >>>>>>> capable >>>>>>> > > hardware >>>>>>> > > > > > seems >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are mostly >>>>>>> passing >>>>>>> > now >>>>>>> > > on >>>>>>> > > > > > > branch-1 >>>>>>> > > > > > > >>> and master. Lets try and keep it this way and start >>>>>>> to trust >>>>>>> > > our >>>>>>> > > > > > test >>>>>>> > > > > > > runs >>>>>>> > > > > > > >>> again. Just a few flakies remain. Lets try and >>>>>>> nail them. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test suites >>>>>>> where >>>>>>> > > > previous >>>>>>> > > > > > we >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our zombie >>>>>>> > detector >>>>>>> > > > > > reports >>>>>>> > > > > > > >>> tests from another project altogether as lingerers >>>>>>> (To be >>>>>>> > > fixed). >>>>>>> > > > > > > Some of >>>>>>> > > > > > > >>> our tests are failing because a concurrent hbase run >>>>>>> is >>>>>>> > undoing >>>>>>> > > > > > > classes and >>>>>>> > > > > > > >>> data from under it. Also, lets fix. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for them to >>>>>>> > complete. >>>>>>> > > > > Many >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up multiple >>>>>>> > clusters >>>>>>> > > > and >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle they >>>>>>> pass at >>>>>>> > all. >>>>>>> > > > > > > Usually >>>>>>> > > > > > > >>> integration tests have been cast as unit tests >>>>>>> because there >>>>>>> > > was >>>>>>> > > > no >>>>>>> > > > > > > where >>>>>>> > > > > > > >>> else for them to get an airing. We have the >>>>>>> hbase-it suite >>>>>>> > now >>>>>>> > > > > which >>>>>>> > > > > > > would >>>>>>> > > > > > > >>> be a more apt place but until these are run on a >>>>>>> regular >>>>>>> > basis >>>>>>> > > in >>>>>>> > > > > > > public >>>>>>> > > > > > > >>> for all to see, the fat integration tests disguised >>>>>>> as unit >>>>>>> > > tests >>>>>>> > > > > > will >>>>>>> > > > > > > >>> remain. A review of our current unit tests weeding >>>>>>> the old >>>>>>> > > cruft >>>>>>> > > > > and >>>>>>> > > > > > > the >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a nice >>>>>>> undertaking >>>>>>> > if >>>>>>> > > > > > > someone is >>>>>>> > > > > > > >>> looking to contribute. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> Alex Newman has been working on making our tests >>>>>>> work up on >>>>>>> > > > travis >>>>>>> > > > > > and >>>>>>> > > > > > > >>> circle-ci. That'll be sweet when it goes >>>>>>> end-to-end. He >>>>>>> > also >>>>>>> > > > > added >>>>>>> > > > > > in >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter, >>>>>>> mapreduce -- >>>>>>> > > > > alongside >>>>>>> > > > > > > our >>>>>>> > > > > > > >>> old "sizing" categorizations of small/medium/large. >>>>>>> His >>>>>>> > > thinking >>>>>>> > > > > is >>>>>>> > > > > > > that >>>>>>> > > > > > > >>> we can run these categorizations in parallel so we >>>>>>> could run >>>>>>> > > the >>>>>>> > > > > > total >>>>>>> > > > > > > >>> suite in about the time of the longest test, say >>>>>>> > 20-30minutes? >>>>>>> > > > We >>>>>>> > > > > > > could >>>>>>> > > > > > > >>> even change Apache to run them this way. >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> FYI, >>>>>>> > > > > > > >>> St.Ack >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >>> >>>>>>> > > > > > > >> >>>>>>> > > > > > > > >>>>>>> > > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > -- >>>>>>> > > > > > Sean >>>>>>> > > > > > >>>>>>> > > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > -- >>>>>>> > > > Best regards, >>>>>>> > > > >>>>>>> > > > - Andy >>>>>>> > > > >>>>>>> > > > Problems worthy of attack prove their worth by hitting back. - >>>>>>> Piet >>>>>>> > Hein >>>>>>> > > > (via Tom White) >>>>>>> > > > >>>>>>> > > >>>>>>> > >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sean >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean >>>> >>> >>> >>> >>> -- >>> Sean >>> >> >> >> >> -- >> Sean >> > > > > -- > Sean > -- Sean
