Stuff seems to be basically working. I just committed a fix for this message so it should be gone now:
grep: unknown devices method grep: write error: Broken pipe Hopefully we've seen last of the errant surefire kills. We have an HBASE_TRUNK build and we have an HBASE_TRUNK_MATRIX. We should get rid of the former now? St.Ack On Mon, Nov 2, 2015 at 11:03 PM, Stack <st...@duboce.net> wrote: > I just did an edit of all of our jenkins configuration. I changed the > post-build zombie section. The surefire-killer was hanging out here it > seems (For detail, see HBASE-14589 Looking for the surefire-killer; builds > being killed...). I may have messed up builds. Will fix in morning if so > (currently builds seem a bit backed up so won't wait around). > > St.Ack > > On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <bus...@cloudera.com> wrote: > >> Hi Folks, >> >> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS] >> reducing our load on builds.a.o' on reasoning or if you have concerns. >> >> Our project job view now has a brief summary of what's going on: >> >> https://builds.apache.org/view/H-L/view/HBase/ >> >> * Where we had matrix builds, I confirmed they're doing the same thing >> (in at least one test-configuration) as the non-matrix version >> * I disabled jobs that were duplicating work done in matrix jobs, or >> that only worked on a tag >> * Made sure all non-disabled 0.98+ jobs are set to notify jira + >> builds mailing list >> * I updated pre-1.1 jobs to look for changes 1/day >> * I updated 1.1 jobs to look for changes every 8 hours >> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours >> >> There's still some non-pressing clean up work to do, so I'll file a >> jira later today just to make sure it's all recorded somewhere. >> >> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <bus...@cloudera.com> >> wrote: >> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2 >> builds to >> > use branch-1.2. (ref HBASE-13912) >> > >> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <bus...@cloudera.com> >> wrote: >> >> >> >> I've add a new build test to run the ITs under sunny day conditions >> using >> >> minicluster for both java 7 and java 8 on the 1.2 line. >> >> >> >> https://builds.apache.org/job/HBase-1.2-IT/ >> >> >> >> I haven't turned on notifications yet, because BulkIngest and >> TestIngest >> >> are read. details on HBASE-13750 >> >> >> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <bus...@cloudera.com> >> wrote: >> >>> >> >>> sigh. that should have been "is now a matrix build". >> >>> >> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <bus...@cloudera.com> >> wrote: >> >>>> >> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8 >> in >> >>>> parallel. >> >>>> >> >>>> I saved the old job for now and left it disabled[2]. >> >>>> >> >>>> >> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/ >> >>>> [2]: >> >>>> >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/ >> >>>> >> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <bus...@cloudera.com> >> >>>> wrote: >> >>>>> >> >>>>> Sorry for the noise. I also updated the checkstyle step on >> HBase-TRUNK >> >>>>> >> >>>>> from >> >>>>> mvn checkstyle:checkstyle >> >>>>> to >> >>>>> mvn -DskipTests package checkstyle:checkstyle >> >>>>> >> >>>>> to deal with the same issue. Looks all clear now. >> >>>>> >> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <bus...@cloudera.com> >> >>>>> wrote: >> >>>>>> >> >>>>>> Updated the following builds: >> >>>>>> >> >>>>>> * HBase-TRUNK >> >>>>>> >> >>>>>> moved from >> >>>>>> >> >>>>>> mvn clean compile findbugs:findbugs >> >>>>>> >> >>>>>> to >> >>>>>> >> >>>>>> mvn clean -DskipTests package findbugs:findbugs >> >>>>>> >> >>>>>> To work around the known issue where we can't do a bootstrap >> compile >> >>>>>> without previous install or remote SNAPSHOT artifacts. (recently >> triggered >> >>>>>> by the addition of the procedure module on master) >> >>>>>> >> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino...@gmail.com> wrote: >> >>>>>>> >> >>>>>>> Make findbugs and checkstyle plugins always run. >> >>>>>>> The default behavior only publishes result for stable builds, but >> >>>>>>> master is >> >>>>>>> not always stable and sometimes we introduce new warnings in >> unstable >> >>>>>>> builds(see builds 6310-6314). >> >>>>>>> And findbugs and checkstyle will not fail unless we abort the >> >>>>>>> building >> >>>>>>> process I think, so it is safe to turn it on every time. >> >>>>>>> >> >>>>>>> Thanks. >> >>>>>>> >> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: >> >>>>>>> >> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report. >> >>>>>>> > >> >>>>>>> > Cheers >> >>>>>>> > >> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino...@gmail.com> >> wrote: >> >>>>>>> > >> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show >> >>>>>>> > > findbugs, >> >>>>>>> > > checkstyle and jacoco coverage report. >> >>>>>>> > > The config has been tested on >> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly >> 30 >> >>>>>>> > > times. >> >>>>>>> > Not >> >>>>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(the >> >>>>>>> > > overhead >> >>>>>>> > of >> >>>>>>> > > collecting information for code coverage). >> >>>>>>> > > Thanks. >> >>>>>>> > > >> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurt...@apache.org >> >: >> >>>>>>> > > >> >>>>>>> > > > +1, thanks a lot for improving our build hygiene. >> >>>>>>> > > > >> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar >> >>>>>>> > > > <enis....@gmail.com> >> >>>>>>> > > wrote: >> >>>>>>> > > > >> >>>>>>> > > > > Thanks Sean. This is great. >> >>>>>>> > > > > >> >>>>>>> > > > > Enis >> >>>>>>> > > > > >> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey >> >>>>>>> > > > > <bus...@cloudera.com> >> >>>>>>> > > > wrote: >> >>>>>>> > > > > >> >>>>>>> > > > > > FYI I just finished chasing down the breakage for mvn >> site >> >>>>>>> > > > > > on all >> >>>>>>> > > patch >> >>>>>>> > > > > > builds. >> >>>>>>> > > > > > >> >>>>>>> > > > > > HBASE-13191 consolidates the few places in the >> test-patch >> >>>>>>> > > > > > code >> >>>>>>> > where >> >>>>>>> > > we >> >>>>>>> > > > > > hard coded MAVEN_OPTS. >> >>>>>>> > > > > > >> >>>>>>> > > > > > If you look at the PreCommit job now, we use the "set >> >>>>>>> > > > > > environment >> >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then everything >> >>>>>>> > > > > > else >> >>>>>>> > respects >> >>>>>>> > > > > that >> >>>>>>> > > > > > setting. >> >>>>>>> > > > > > >> >>>>>>> > > > > > I set the initial value to be a combination of the >> memory >> >>>>>>> > limitations >> >>>>>>> > > > > we've >> >>>>>>> > > > > > been actually running with (the ~6G was getting ignored) >> >>>>>>> > > > > > and the >> >>>>>>> > > > permgen >> >>>>>>> > > > > > needed for site. >> >>>>>>> > > > > > >> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData >> -XX:MaxPermSize=256m >> >>>>>>> > > > > > >> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack < >> st...@duboce.net> >> >>>>>>> > > > > > wrote: >> >>>>>>> > > > > > >> >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm as >> >>>>>>> > > > > > > patch-build >> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS >> to >> >>>>>>> > > > > > > be the >> >>>>>>> > > same >> >>>>>>> > > > as >> >>>>>>> > > > > > > those of trunk build too, setting >> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"... it >> >>>>>>> > had >> >>>>>>> > > > been >> >>>>>>> > > > > > > 3000. >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > Yours, >> >>>>>>> > > > > > > St.Ack >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack < >> st...@duboce.net> >> >>>>>>> > > > > > > wrote: >> >>>>>>> > > > > > > >> >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds >> and >> >>>>>>> > > > > > > > or last >> >>>>>>> > 7 >> >>>>>>> > > > > days, >> >>>>>>> > > > > > > > whichever comes first. >> >>>>>>> > > > > > > > St.Ack >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack >> >>>>>>> > > > > > > > <st...@duboce.net> >> >>>>>>> > wrote: >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run >> mostly >> >>>>>>> > > > > > > >> blue >> >>>>>>> > > (give >> >>>>>>> > > > or >> >>>>>>> > > > > > > take >> >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue >> branch-1 >> >>>>>>> > > > > > > >> has >> >>>>>>> > > helped >> >>>>>>> > > > us >> >>>>>>> > > > > > > >> identify at least one destabilizing commit in the >> last >> >>>>>>> > > > > > > >> few >> >>>>>>> > days, >> >>>>>>> > > > > maybe >> >>>>>>> > > > > > > two; >> >>>>>>> > > > > > > >> this is as it should be (smile). >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch, >> make >> >>>>>>> > > > > > > >> sure >> >>>>>>> > > > > subsequent >> >>>>>>> > > > > > > >> builds stay blue. You can subscribe to >> >>>>>>> > bui...@hbase.apache.org >> >>>>>>> > > to >> >>>>>>> > > > > get >> >>>>>>> > > > > > > >> notice of failures if not already subscribed. >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> Thanks, >> >>>>>>> > > > > > > >> St.Ack >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> 1. >> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/ >> >>>>>>> > > > > > > >> 2. >> >>>>>>> > > >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/ >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack >> >>>>>>> > > > > > > >> <st...@duboce.net> >> >>>>>>> > > wrote: >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > >>> A few notes on testing. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now and >> after >> >>>>>>> > > > > > > >>> some >> >>>>>>> > > work, >> >>>>>>> > > > we >> >>>>>>> > > > > > are >> >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. >> Lets >> >>>>>>> > > > > > > >>> try and >> >>>>>>> > > keep >> >>>>>>> > > > it >> >>>>>>> > > > > > > this >> >>>>>>> > > > > > > >>> way going forward. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with more >> >>>>>>> > > > > > > >>> capable >> >>>>>>> > > hardware >> >>>>>>> > > > > > seems >> >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are mostly >> >>>>>>> > > > > > > >>> passing >> >>>>>>> > now >> >>>>>>> > > on >> >>>>>>> > > > > > > branch-1 >> >>>>>>> > > > > > > >>> and master. Lets try and keep it this way and >> start >> >>>>>>> > > > > > > >>> to trust >> >>>>>>> > > our >> >>>>>>> > > > > > test >> >>>>>>> > > > > > > runs >> >>>>>>> > > > > > > >>> again. Just a few flakies remain. Lets try and >> nail >> >>>>>>> > > > > > > >>> them. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test >> suites >> >>>>>>> > > > > > > >>> where >> >>>>>>> > > > previous >> >>>>>>> > > > > > we >> >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our >> zombie >> >>>>>>> > detector >> >>>>>>> > > > > > reports >> >>>>>>> > > > > > > >>> tests from another project altogether as lingerers >> >>>>>>> > > > > > > >>> (To be >> >>>>>>> > > fixed). >> >>>>>>> > > > > > > Some of >> >>>>>>> > > > > > > >>> our tests are failing because a concurrent hbase >> run >> >>>>>>> > > > > > > >>> is >> >>>>>>> > undoing >> >>>>>>> > > > > > > classes and >> >>>>>>> > > > > > > >>> data from under it. Also, lets fix. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for >> them to >> >>>>>>> > complete. >> >>>>>>> > > > > Many >> >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up >> multiple >> >>>>>>> > clusters >> >>>>>>> > > > and >> >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle they >> >>>>>>> > > > > > > >>> pass at >> >>>>>>> > all. >> >>>>>>> > > > > > > Usually >> >>>>>>> > > > > > > >>> integration tests have been cast as unit tests >> >>>>>>> > > > > > > >>> because there >> >>>>>>> > > was >> >>>>>>> > > > no >> >>>>>>> > > > > > > where >> >>>>>>> > > > > > > >>> else for them to get an airing. We have the >> hbase-it >> >>>>>>> > > > > > > >>> suite >> >>>>>>> > now >> >>>>>>> > > > > which >> >>>>>>> > > > > > > would >> >>>>>>> > > > > > > >>> be a more apt place but until these are run on a >> >>>>>>> > > > > > > >>> regular >> >>>>>>> > basis >> >>>>>>> > > in >> >>>>>>> > > > > > > public >> >>>>>>> > > > > > > >>> for all to see, the fat integration tests >> disguised >> >>>>>>> > > > > > > >>> as unit >> >>>>>>> > > tests >> >>>>>>> > > > > > will >> >>>>>>> > > > > > > >>> remain. A review of our current unit tests >> weeding >> >>>>>>> > > > > > > >>> the old >> >>>>>>> > > cruft >> >>>>>>> > > > > and >> >>>>>>> > > > > > > the >> >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a nice >> >>>>>>> > > > > > > >>> undertaking >> >>>>>>> > if >> >>>>>>> > > > > > > someone is >> >>>>>>> > > > > > > >>> looking to contribute. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> Alex Newman has been working on making our tests >> work >> >>>>>>> > > > > > > >>> up on >> >>>>>>> > > > travis >> >>>>>>> > > > > > and >> >>>>>>> > > > > > > >>> circle-ci. That'll be sweet when it goes >> end-to-end. >> >>>>>>> > > > > > > >>> He >> >>>>>>> > also >> >>>>>>> > > > > added >> >>>>>>> > > > > > in >> >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter, >> >>>>>>> > > > > > > >>> mapreduce -- >> >>>>>>> > > > > alongside >> >>>>>>> > > > > > > our >> >>>>>>> > > > > > > >>> old "sizing" categorizations of >> small/medium/large. >> >>>>>>> > > > > > > >>> His >> >>>>>>> > > thinking >> >>>>>>> > > > > is >> >>>>>>> > > > > > > that >> >>>>>>> > > > > > > >>> we can run these categorizations in parallel so we >> >>>>>>> > > > > > > >>> could run >> >>>>>>> > > the >> >>>>>>> > > > > > total >> >>>>>>> > > > > > > >>> suite in about the time of the longest test, say >> >>>>>>> > 20-30minutes? >> >>>>>>> > > > We >> >>>>>>> > > > > > > could >> >>>>>>> > > > > > > >>> even change Apache to run them this way. >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> FYI, >> >>>>>>> > > > > > > >>> St.Ack >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >>> >> >>>>>>> > > > > > > >> >> >>>>>>> > > > > > > > >> >>>>>>> > > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > -- >> >>>>>>> > > > > > Sean >> >>>>>>> > > > > > >> >>>>>>> > > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > -- >> >>>>>>> > > > Best regards, >> >>>>>>> > > > >> >>>>>>> > > > - Andy >> >>>>>>> > > > >> >>>>>>> > > > Problems worthy of attack prove their worth by hitting >> back. - >> >>>>>>> > > > Piet >> >>>>>>> > Hein >> >>>>>>> > > > (via Tom White) >> >>>>>>> > > > >> >>>>>>> > > >> >>>>>>> > >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Sean >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Sean >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Sean >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Sean >> >> >> >> >> >> >> >> >> >> -- >> >> Sean >> > >> > >> > >> > >> > -- >> > Sean >> >> >> >> -- >> Sean >> > >
