Ouch. Thanks Sean!
I'm pretty sure at some point I was debugging 1.3-IT job and saw branch-1.3
getting checked out in the logs. Not sure how/when it went sideways though.
Yeah, let's see how it goes.
-Mikhail
On Wed, Mar 1, 2017 at 5:50 AM, Sean Busbey <bus...@apache.org> wrote:
> Fun times.
>
> 1) Turns out our 1.3-IT jobs have been running against branch-1.2.
> Don't know how long, but as long as we have history.
>
> 2) I deleted the failing-since-august 1.2-IT job.
>
> 3) I renamed the passing 1.3-IT job that runs against branch-1.2 to be
> the 1.2-IT job
>
> 4) I copied the now renamed 1.2-IT job and made a 1.3-IT job that runs
> against branch-1.3
>
> I kicked off jobs after all this shuffling. We'll see how it goes.
>
> On Tue, Feb 21, 2017 at 5:49 PM, Sean Busbey <bus...@apache.org> wrote:
> > FYI, I updated the 1.2-IT and 1.3-IT jobs today to use Appy's
> > suggested "custom child workspace" of "${SHORT_COMBINATION}", since
> > spaces in paths had caused them to fail for a v long time.
> >
> > On Fri, Oct 14, 2016 at 4:44 PM, Andrew Purtell <apurt...@apache.org>
> wrote:
> >> Thanks Ted, that would be a nice contribution, thank you.
> >>
> >>
> >> On Fri, Oct 14, 2016 at 12:07 PM, Apekshit Sharma <a...@cloudera.com>
> wrote:
> >>
> >>> @Ted, here's the old jira, HBASE-14167. Use that.
> >>>
> >>> On Fri, Oct 14, 2016 at 12:02 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>>
> >>> > I just ran the tests in hbase-spark module using 'mvn verify'.
> >>> >
> >>> > All passed.
> >>> >
> >>> > I am testing a patch locally where hbase-spark tests are run in test
> >>> phase.
> >>> >
> >>> > If the tests pass, I will log a JIRA.
> >>> >
> >>> > Thanks
> >>> >
> >>> > > On Oct 14, 2016, at 11:41 AM, Andrew Purtell <apurt...@apache.org>
> >>> > wrote:
> >>> > >
> >>> > > The hbase-spark integration tests run (and fail) for me locally
> >>> whenever
> >>> > I
> >>> > > build master with 'mvn clean install -DskipITs' .
> >>> > >
> >>> > > HBaseConnectionCacheSuite:
> >>> > > - all test cases *** FAILED ***
> >>> > > 2 did not equal 1 (HBaseConnectionCacheSuite.scala:92)
> >>> > >
> >>> > > Saw it but had to ignore/triage to get something else done.
> >>> > >
> >>> > > We have a weird situation where integration tests run when they
> >>> shouldn't
> >>> > > locally yet no tests run at all for patch process?
> >>> > >
> >>> > > I would like to see Spark behave like the other modules. I remember
> >>> > filing
> >>> > > a JIRA asking that hbase-spark honor -DskipITs. It still doesn't.
> >>> > > Meanwhile, it does its own thing with '-DskipSparkTests', which is
> not
> >>> > > appropriate given that none of the other modules have their own
> >>> distinct
> >>> > > control parameters. There also doesn't seem to be a distinction
> between
> >>> > > unit tests and integration tests. The 'test' target does nothing.
> >>> > > Everything happens during the 'integration-test' phase. Is this a
> Spark
> >>> > > limitation?
> >>> > >
> >>> > >
> >>> > >> On Fri, Oct 14, 2016 at 11:27 AM, Sean Busbey <
> bus...@cloudera.com>
> >>> > wrote:
> >>> > >>
> >>> > >> Do the HBase Spark tests only run during the maven verify command?
> >>> > >> We'll need to update our personality to say that that command
> should
> >>> > >> be used for unit tests when in the hbase spark module. ugh.
> >>> > >>
> >>> > >> On Thu, Oct 13, 2016 at 7:42 PM, Apekshit Sharma <
> a...@cloudera.com>
> >>> > >> wrote:
> >>> > >>> Our patch process isn't running hbase-spark tests. See this for
> >>> > example:
> >>> > >>>
> >>> > >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/
> >>> > >>> https://builds.apache.org/job/PreCommit-HBASE-Build/3842/
> >>> > >> artifact/patchprocess/patch-unit-hbase-spark.txt/*view*/
> >>> > >>>
> >>> > >>> Found it when trying to debug cause of trunk failures. Part of
> the
> >>> > cause
> >>> > >> is
> >>> > >>> hbase-spark's HBaseConnectionCacheSuite test failure (
> >>> > >>> https://builds.apache.org/view/All/job/HBase-Trunk_
> >>> > >> matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/
> 1776/consoleFull
> >>> > >> <https://builds.apache.org/view/All/job/HBase-Trunk_
> >>> > matrix/jdk=JDK%201.8%20%28latest%29,label=yahoo-not-
> h2/1776/consoleFull>
> >>> > >> )
> >>> > >>> which was added in HBASE-16638. However, to be fair, QA was
> green and
> >>> > >>> reported passing hbase-spark tests for that jira.
> >>> > >>>
> >>> > >>>> On Mon, Sep 19, 2016 at 12:57 PM, Stack <st...@duboce.net>
> wrote:
> >>> > >>>>
> >>> > >>>> childCustomWorkspace seems to be just the ticket. Nice find
> Appy.
> >>> > >>>> St.Ack
> >>> > >>>>
> >>> > >>>> On Mon, Sep 19, 2016 at 10:03 AM, Sean Busbey <
> bus...@cloudera.com>
> >>> > >> wrote:
> >>> > >>>>
> >>> > >>>>> Option 2c looks to be working really well. Thanks for tackling
> this
> >>> > >> Appy!
> >>> > >>>>>
> >>> > >>>>> We still have some failures on the master build, but it looks
> like
> >>> > >>>>> actual problems (or perhaps a flakey). There are several
> passing
> >>> > >>>>> builds.
> >>> > >>>>>
> >>> > >>>>> This should be pretty easy to replicate on the other jobs. I
> don't
> >>> > see
> >>> > >>>>> a downside. Anyone else have concerns?
> >>> > >>>>>
> >>> > >>>>>
> >>> > >>>>> On Fri, Sep 16, 2016 at 6:15 PM, Apekshit Sharma <
> >>> a...@cloudera.com>
> >>> > >>>>> wrote:
> >>> > >>>>>> So this all started with spaces-in-path issue, right? I
> think it
> >>> > >> has
> >>> > >>>>>> gobbled up a lot of time of a lot of people.
> >>> > >>>>>> Let's discuss our options and try to fix it for good. Here are
> >>> what
> >>> > >> i
> >>> > >>>> can
> >>> > >>>>>> think of, and my opinion about them.
> >>> > >>>>>>
> >>> > >>>>>> 1. Not use matrix build
> >>> > >>>>>> Temporary fix. Not preferred since not applicable to
> >>> other
> >>> > >>>>>> branches' builds.
> >>> > >>>>>>
> >>> > >>>>>> 2. Use matrix build
> >>> > >>>>>>
> >>> > >>>>>> a. Use tool environment trick
> >>> > >>>>>> I applied this few days ago. Seemed to work until we
> >>> > >>>>> discovered
> >>> > >>>>>> scalatest issue. While the solution looks legitimate, we can't
> >>> trust
> >>> > >>>> that
> >>> > >>>>>> all tools will use JAVA_HOME instead of directly using java
> >>> command.
> >>> > >>>>>>
> >>> > >>>>>> b. Use JDK axix
> >>> > >>>>>> Doesn't work right now. I don't have much idea of
> what's
> >>> > >> the
> >>> > >>>>> cost
> >>> > >>>>>> for fixing it.
> >>> > >>>>>>
> >>> > >>>>>> c. Use JDK axis with custom child workspace
> >>> > >>>>>>
> >>> > >>>>>> https://github.com/jenkinsci/matrix-project-plugin/blob/
> >>> > >>>>> master/src/main/resources/hudson/matrix/MatrixProject/
> >>> > >>>>> help-childCustomWorkspace.html
> >>> > >>>>>> Just found this one, and it might solve things for
> good.
> >>> I
> >>> > >>>> have
> >>> > >>>>>> updated the job to use this. Let's see how it works.
> >>> > >>>>>>
> >>> > >>>>>> What do others think?
> >>> > >>>>>>
> >>> > >>>>>>> On Fri, Sep 16, 2016 at 3:31 PM, Stack <st...@duboce.net>
> wrote:
> >>> > >>>>>>>
> >>> > >>>>>>> The profile (or define) skipSparkTests looks like it will
> skip
> >>> > >> spark
> >>> > >>>>> tests.
> >>> > >>>>>>> Setting skipIntegrationTests to true will skip it.
> >>> > >>>>>>> S
> >>> > >>>>>>>
> >>> > >>>>>>> On Fri, Sep 16, 2016 at 1:40 PM, Dima Spivak <
> >>> > >> dimaspi...@apache.org>
> >>> > >>>>>>> wrote:
> >>> > >>>>>>>
> >>> > >>>>>>>> Doesn't seem we need a matrix project for master anymore
> since
> >>> > >> we're
> >>> > >>>>> just
> >>> > >>>>>>>> doing JDK 8 now, right? Also, it looks like the hbase-spark
> >>> > >>>>>>>> integration-test phase is what's tripping up the build. Why
> not
> >>> > >> just
> >>> > >>>>>>>> temporarily disable that to unblock testing?
> >>> > >>>>>>>>
> >>> > >>>>>>>> On Friday, September 16, 2016, Apekshit Sharma <
> >>> > >> a...@cloudera.com>
> >>> > >>>>>>> wrote:
> >>> > >>>>>>>>
> >>> > >>>>>>>>> So the issue is, we set JAVA_HOME to jdk8 based on matrix
> param
> >>> > >>>> and
> >>> > >>>>>>> using
> >>> > >>>>>>>>> tool environment. Since mvn uses the env variable, it
> compiles
> >>> > >>>> with
> >>> > >>>>> jdk
> >>> > >>>>>>>> 8.
> >>> > >>>>>>>>> But i suspect that scalatest isn't using the env variable,
> >>> > >> instead
> >>> > >>>>> it
> >>> > >>>>>>>> might
> >>> > >>>>>>>>> be directly using 'java' cmd, which can be jdk 7 or 8, and
> can
> >>> > >>>> vary
> >>> > >>>>> by
> >>> > >>>>>>>>> machine.
> >>> > >>>>>>>>> Build succeed if 'java' points to jdk 8, otherwise fails.
> >>> > >>>>>>>>> Note that we didn't have this issue earlier since we were
> using
> >>> > >>>>> jenkins
> >>> > >>>>>>>>> 'JDK' axis which would set the 'java' to the appropriate
> >>> > >> version.
> >>> > >>>>> But
> >>> > >>>>>>>> that
> >>> > >>>>>>>>> methods had spaces-in-path issue, so i had to change it.
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>
> >>> > >>>>>>>>> On Fri, Sep 16, 2016 at 3:46 AM, aman poonia <
> >>> > >>>>> aman.poonia...@gmail.com
> >>> > >>>>>>>>> <javascript:;>>
> >>> > >>>>>>>>> wrote:
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>> I am not sure if this will help. But it looks like it is
> >>> > >> because
> >>> > >>>>> of
> >>> > >>>>>>>>> version
> >>> > >>>>>>>>>> mismatch, that is, it is expecting JDK1.7 and we are
> >>> > >> compiling
> >>> > >>>>> with
> >>> > >>>>>>>>> jdk1.8.
> >>> > >>>>>>>>>> That means there is some library which has to be compiled
> >>> > >> with
> >>> > >>>>> jdk8
> >>> > >>>>>>> or
> >>> > >>>>>>>>>> needs to be updated to a jdk8 compatible version.
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>> --
> >>> > >>>>>>>>>> *With Regards:-*
> >>> > >>>>>>>>>> *Aman Poonia*
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>> On Fri, Sep 16, 2016 at 2:40 AM, Apekshit Sharma <
> >>> > >>>>> a...@cloudera.com
> >>> > >>>>>>>>> <javascript:;>>
> >>> > >>>>>>>>>> wrote:
> >>> > >>>>>>>>>>
> >>> > >>>>>>>>>>> And....everything is back to red.
> >>> > >>>>>>>>>>> Because something is plaguing our builds again. :(
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>> If anyone knows what's problem in this case, please reply
> >>> > >> on
> >>> > >>>>> this
> >>> > >>>>>>>>> thread,
> >>> > >>>>>>>>>>> otherwise i'll try to fix it later sometime today.
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>> [INFO] *--- scalatest-maven-plugin:1.0:test
> >>> > >>>> (integration-test)
> >>> > >>>>> @
> >>> > >>>>>>>>>>> hbase-spark ---
> >>> > >>>>>>>>>>> * [36mDiscovery starting. [0m
> >>> > >>>>>>>>>>> [31m*** RUN ABORTED *** [0m
> >>> > >>>>>>>>>>> [31m java.lang.UnsupportedClassVersionError:
> >>> > >>>>>>>>>>> org/apache/hadoop/hbase/spark/example/hbasecontext/
> >>> > >>>>>>>>>>> JavaHBaseDistributedScan
> >>> > >>>>>>>>>>> : Unsupported major.minor version 52.0 [0m
> >>> > >>>>>>>>>>> [31m at java.lang.ClassLoader.defineClass1(Native
> >>> > >> Method)
> >>> > >>>> [0m
> >>> > >>>>>>>>>>> [31m at java.lang.ClassLoader.
> >>> > >> defineClass(ClassLoader.java:
> >>> > >>>>> 803)
> >>> > >>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.security.SecureClassLoader.defineClass(
> >>> > >>>>>>>>>>> SecureClassLoader.java:142)
> >>> > >>>>>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.net.URLClassLoader.
> >>> > >> defineClass(URLClassLoader.
> >>> > >>>>>>>> java:449)
> >>> > >>>>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.net.URLClassLoader.
> >>> > >> access$100(URLClassLoader.
> >>> > >>>>>>> java:71)
> >>> > >>>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.net.URLClassLoader$1.run(
> >>> > >>>>> URLClassLoader.java:361)
> >>> > >>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.net.URLClassLoader$1.run(
> >>> > >>>>> URLClassLoader.java:355)
> >>> > >>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.security.AccessController.doPrivileged(
> >>> > >> Native
> >>> > >>>>>>> Method)
> >>> > >>>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.net.URLClassLoader.
> >>> > >>>>> findClass(URLClassLoader.java:
> >>> > >>>>>>> 354)
> >>> > >>>>>>>>> [0m
> >>> > >>>>>>>>>>> [31m at java.lang.ClassLoader.
> >>> > >> loadClass(ClassLoader.java:
> >>> > >>>> 425)
> >>> > >>>>>>> [0m
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>> On Mon, Sep 12, 2016 at 5:01 PM, Mikhail Antonov <
> >>> > >>>>>>>> olorinb...@gmail.com
> >>> > >>>>>>>>> <javascript:;>>
> >>> > >>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>> Great work indeed!
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>> Agreed, occasional failed runs may not be that bad, but
> >>> > >>>> fairly
> >>> > >>>>>>>>> regular
> >>> > >>>>>>>>>>>> failed runs ruin the idea of CI. Especially for released
> >>> > >> or
> >>> > >>>>>>>> otherwise
> >>> > >>>>>>>>>>>> supposedly stable branches.
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>> -Mikhail
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>> On Mon, Sep 12, 2016 at 4:53 PM, Sean Busbey <
> >>> > >>>>>>> bus...@cloudera.com
> >>> > >>>>>>>>> <javascript:;>>
> >>> > >>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>> awesome work Appy!
> >>> > >>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>> That's certainly good news to hear.
> >>> > >>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>> On Mon, Sep 12, 2016 at 2:14 PM, Apekshit Sharma <
> >>> > >>>>>>>>> a...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>> On a separate note:
> >>> > >>>>>>>>>>>>>> Trunk had 8 green runs in last 3 days! (
> >>> > >>>>>>>>>>>>>> https://builds.apache.org/job/HBase-Trunk_matrix/)
> >>> > >>>>>>>>>>>>>> This was due to fixing just the mass failures on
> >>> > >> trunk
> >>> > >>>>> and no
> >>> > >>>>>>>>>> change
> >>> > >>>>>>>>>>> in
> >>> > >>>>>>>>>>>>>> flaky infra. Which made me to conclude two things:
> >>> > >>>>>>>>>>>>>> 1. Flaky infra works.
> >>> > >>>>>>>>>>>>>> 2. It relies heavily on the post-commit build's
> >>> > >>>> stability
> >>> > >>>>>>>> (which
> >>> > >>>>>>>>>>> every
> >>> > >>>>>>>>>>>>>> project should anyways strive for). If the build
> >>> > >> fails
> >>> > >>>>>>>>>>> catastrophically
> >>> > >>>>>>>>>>>>>> once in a while, we can just exclude that one run
> >>> > >> using
> >>> > >>>> a
> >>> > >>>>>>> flag
> >>> > >>>>>>>>> and
> >>> > >>>>>>>>>>>>>> everything will work, but if it happens frequently,
> >>> > >> then
> >>> > >>>>> it
> >>> > >>>>>>>> won't
> >>> > >>>>>>>>>>> work
> >>> > >>>>>>>>>>>>>> right.
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>> I have re-enabled Flaky tests job (
> >>> > >>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>> view/H-L/view/HBase/job/HBASE-
> >>> > >>>>>>>>>> Flaky-Tests/
> >>> > >>>>>>>>>>> )
> >>> > >>>>>>>>>>>>> which
> >>> > >>>>>>>>>>>>>> was disabled for almost a month due to trunk being on
> >>> > >>>>> fire.
> >>> > >>>>>>>>>>>>>> I will keep an eye on how things are going.
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>> On Mon, Sep 12, 2016 at 2:02 PM, Apekshit Sharma <
> >>> > >>>>>>>>>> a...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>> @Sean, Mikhail: I found the alternate solution.
> >>> > >> Using
> >>> > >>>>> user
> >>> > >>>>>>>>> defined
> >>> > >>>>>>>>>>>> axis,
> >>> > >>>>>>>>>>>>>>> tool environment and env variable injection.
> >>> > >>>>>>>>>>>>>>> See latest diff to https://builds.apache.org/job/
> >>> > >>>>>>>>>>> HBase-Trunk_matrix/
> >>> > >>>>>>>>>>>>> job
> >>> > >>>>>>>>>>>>>>> for reference.
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>> On Tue, Aug 30, 2016 at 7:39 PM, Mikhail Antonov <
> >>> > >>>>>>>>>>>> olorinb...@gmail.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> FYI, I did the same for branch-1.3 builds. I've
> >>> > >>>>> disabled
> >>> > >>>>>>>>>> hbase-1.3
> >>> > >>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>> hbase-1.3-IT jobs and instead created
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> https://builds.apache.org/job/HBase-1.3-JDK8 and
> >>> > >>>>>>>>>>>>>>>> https://builds.apache.org/job/HBase-1.3-JDK7
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> This should work for now until we figure out how to
> >>> > >>>> move
> >>> > >>>>>>>>> forward.
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> Thanks,
> >>> > >>>>>>>>>>>>>>>> Mikhail
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> On Wed, Aug 17, 2016 at 1:41 PM, Sean Busbey <
> >>> > >>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> /me smacks forehead
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> these replacement jobs, of course, also have
> >>> > >> special
> >>> > >>>>>>>>> characters
> >>> > >>>>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>> their names which then show up in the working
> >>> > >> path.
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> renaming them to skip spaces and parens.
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> On Wed, Aug 17, 2016 at 1:34 PM, Sean Busbey <
> >>> > >>>>>>>>>>>> sean.bus...@gmail.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>> FYI, it looks like essentially our entire CI
> >>> > >> suite
> >>> > >>>>> is
> >>> > >>>>>>>> red,
> >>> > >>>>>>>>>>>> probably
> >>> > >>>>>>>>>>>>>>>> due
> >>> > >>>>>>>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>> parts of our codebase not tolerating spaces or
> >>> > >>>> other
> >>> > >>>>>>>>> special
> >>> > >>>>>>>>>>>>>>>> characters
> >>> > >>>>>>>>>>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>>> the working directory.
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> I've made a stop-gap non-multi-configuration
> >>> > >> set
> >>> > >>>> of
> >>> > >>>>>>> jobs
> >>> > >>>>>>>>> for
> >>> > >>>>>>>>>>>>> running
> >>> > >>>>>>>>>>>>>>>> unit
> >>> > >>>>>>>>>>>>>>>>>> tests for the 1.2 branch against JDK 7 and JDK
> >>> > >> 8:
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>>>>> view/H-L/view/HBase/job/HBase%
> >>> > >>>>>>>>>>>>>>>>> 201.2%20(JDK%201.7)/
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>>>>> view/H-L/view/HBase/job/HBase%
> >>> > >>>>>>>>>>>>>>>>> 201.2%20(JDK%201.8)/
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> Due to the lack of response from infra@ I
> >>> > >> suspect
> >>> > >>>>> our
> >>> > >>>>>>>> only
> >>> > >>>>>>>>>>>> options
> >>> > >>>>>>>>>>>>>>>> for
> >>> > >>>>>>>>>>>>>>>>>> continuing on ASF infra is to fix whatever
> >>> > >> part of
> >>> > >>>>> our
> >>> > >>>>>>>>> build
> >>> > >>>>>>>>>>>>> doesn't
> >>> > >>>>>>>>>>>>>>>>>> tolerate the new paths, or stop using
> >>> > >>>>>>> multiconfiguration
> >>> > >>>>>>>>>>>>> deployments.
> >>> > >>>>>>>>>>>>>>>> I
> >>> > >>>>>>>>>>>>>>>>> am
> >>> > >>>>>>>>>>>>>>>>>> obviously less than thrilled at the idea of
> >>> > >> having
> >>> > >>>>>>>> several
> >>> > >>>>>>>>>>>>> multiples
> >>> > >>>>>>>>>>>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>> current jobs.
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> On Wed, Aug 10, 2016 at 6:28 PM, Sean Busbey <
> >>> > >>>>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> Ugh.
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> I sent a reply to Gav on builds@ about maybe
> >>> > >>>>> getting
> >>> > >>>>>>>>> names
> >>> > >>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>> don't
> >>> > >>>>>>>>>>>>>>>>>>> have spaces in them:
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread.html/
> >>> > >>>>>>>>>>>>> 8ac03dc62f9d6862d4f3d5eb37119c
> >>> > >>>>>>>>>>>>>>>>>>> 9c73b4059aaa3ebba52fc63bb6@%3C
> >>> > >> builds.apache.org
> >>> > >>>> %3E
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> In the mean time, is this an issue we need
> >>> > >> file
> >>> > >>>>> with
> >>> > >>>>>>>>> Hadoop
> >>> > >>>>>>>>>> or
> >>> > >>>>>>>>>>>>>>>>>>> something we need to fix in our own code?
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> On Wed, Aug 10, 2016 at 6:04 PM, Matteo
> >>> > >> Bertozzi
> >>> > >>>>>>>>>>>>>>>>>>> <theo.berto...@gmail.com <javascript:;>>
> >>> > >> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>> There are a bunch of builds that have most
> >>> > >> of
> >>> > >>>> the
> >>> > >>>>>>> test
> >>> > >>>>>>>>>>>> failing.
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>> Example:
> >>> > >>>>>>>>>>>>>>>>>>>> https://builds.apache.org/job/
> >>> > >>>>>>>>>> HBase-Trunk_matrix/1392/jdk=
> >>> > >>>>>>>>>>>>>>>>>>> JDK%201.7%20(latest),label=
> >>> > >>>>>>>> yahoo-not-h2/testReport/junit/
> >>> > >>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase/
> >>> > >> TestLocalHBaseCluster/
> >>> > >>>>>>>>>>>>> testLocalHBaseCluster/
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>> from the stack trace looks like the problem
> >>> > >> is
> >>> > >>>>> with
> >>> > >>>>>>>> the
> >>> > >>>>>>>>>> jdk
> >>> > >>>>>>>>>>>> name
> >>> > >>>>>>>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>>> has
> >>> > >>>>>>>>>>>>>>>>>>>> spaces:
> >>> > >>>>>>>>>>>>>>>>>>>> the hadoop FsVolumeImpl calls
> >>> > >>>> setNameFormat(... +
> >>> > >>>>>>>>>>>>>>>> fileName.toString()
> >>> > >>>>>>>>>>>>>>>>> +
> >>> > >>>>>>>>>>>>>>>>>>> ...)
> >>> > >>>>>>>>>>>>>>>>>>>> and this seems to not be escaped
> >>> > >>>>>>>>>>>>>>>>>>>> so we end up with JDK%25201.7%2520(latest)
> >>> > >> in
> >>> > >>>> the
> >>> > >>>>>>>> string
> >>> > >>>>>>>>>>>> format
> >>> > >>>>>>>>>>>>>>>> and we
> >>> > >>>>>>>>>>>>>>>>>>> get
> >>> > >>>>>>>>>>>>>>>>>>>> a IllegalFormatPrecisionException: 7
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>> 2016-08-10 22:07:46,108 WARN [DataNode:
> >>> > >>>>>>>>>>>>>>>>>>>> [[[DISK]file:/home/jenkins/
> >>> > >>>>>>>>> jenkins-slave/workspace/HBase-
> >>> > >>>>>>>>>>>>>>>>>>> Trunk_matrix/jdk/JDK%25201.7%
> >>> > >>>>>>>>> 2520(latest)/label/yahoo-not-
> >>> > >>>>>>>>>>>>>>>>>>> h2/hbase-server/target/test-
> >>> > >>>>>>>> data/e7099624-ecfa-4674-87de-
> >>> > >>>>>>>>>>>>>>>>>>> a8733d13b582/dfscluster_
> >>> > >> 10fdcfc3-cd1b-45be-9b5a-
> >>> > >>>>>>>>>>>>>>>>>>> 9c88f385e6f1/dfs/data/data1/,
> >>> > >>>>>>>>>>>>>>>>>>>> [DISK]file:/home/jenkins/
> >>> > >>>>>>>> jenkins-slave/workspace/HBase-
> >>> > >>>>>>>>>>>>>>>>>>> Trunk_matrix/jdk/JDK%25201.7%
> >>> > >>>>>>>>> 2520(latest)/label/yahoo-not-
> >>> > >>>>>>>>>>>>>>>>>>> h2/hbase-server/target/test-
> >>> > >>>>>>>> data/e7099624-ecfa-4674-87de-
> >>> > >>>>>>>>>>>>>>>>>>> a8733d13b582/dfscluster_
> >>> > >> 10fdcfc3-cd1b-45be-9b5a-
> >>> > >>>>>>>>>>>>>>>>>>> 9c88f385e6f1/dfs/data/data2/]]
> >>> > >>>>>>>>>>>>>>>>>>>> heartbeating to localhost/127.0.0.1:34629]
> >>> > >>>>>>>>>>>>>>>>>>>> datanode.BPServiceActor(831): Unexpected
> >>> > >>>>> exception
> >>> > >>>>>>> in
> >>> > >>>>>>>>>> block
> >>> > >>>>>>>>>>>> pool
> >>> > >>>>>>>>>>>>>>>> Block
> >>> > >>>>>>>>>>>>>>>>>>>> pool <registering> (Datanode Uuid
> >>> > >> unassigned)
> >>> > >>>>>>> service
> >>> > >>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>> localhost/127.0.0.1:34629
> >>> > >>>>>>>>>>>>>>>>>>>> java.util.IllegalFormatPrecisionException:
> >>> > >> 7
> >>> > >>>>>>>>>>>>>>>>>>>> at java.util.Formatter$
> >>> > >>>>>>>>> FormatSpecifier.checkText(
> >>> > >>>>>>>>>>>>>>>>>>> Formatter.java:2984)
> >>> > >>>>>>>>>>>>>>>>>>>> at java.util.Formatter$
> >>> > >>>>>>>> FormatSpecifier.<init>(
> >>> > >>>>>>>>>>>>>>>>>>> Formatter.java:2688)
> >>> > >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.parse(
> >>> > >>>>>>>>> Formatter.java:2528)
> >>> > >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.format(
> >>> > >>>>>>>>>> Formatter.java:2469)
> >>> > >>>>>>>>>>>>>>>>>>>> at java.util.Formatter.format(
> >>> > >>>>>>>>>> Formatter.java:2423)
> >>> > >>>>>>>>>>>>>>>>>>>> at java.lang.String.format(
> >>> > >>>>>>> String.java:2792)
> >>> > >>>>>>>>>>>>>>>>>>>> at com.google.common.util.
> >>> > >> concurrent.
> >>> > >>>>>>>>>>>>> ThreadFactoryBuilder.
> >>> > >>>>>>>>>>>>>>>>>>> setNameFormat(ThreadFactoryBuilder.java:68)
> >>> > >>>>>>>>>>>>>>>>>>>> at org.apache.hadoop.hdfs.server.
> >>> > >>>>>>>>>>>>> datanode.fsdataset.impl.
> >>> > >>>>>>>>>>>>>>>>>>> FsVolumeImpl.initializeCacheExecutor(
> >>> > >>>>>>>>> FsVolumeImpl.java:140)
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>> Matteo
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>> On Tue, Aug 9, 2016 at 9:55 AM, Stack <
> >>> > >>>>>>>> st...@duboce.net
> >>> > >>>>>>>>> <javascript:;>>
> >>> > >>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>> Good on you Sean.
> >>> > >>>>>>>>>>>>>>>>>>>>> S
> >>> > >>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>> On Mon, Aug 8, 2016 at 9:43 PM, Sean
> >>> > >> Busbey <
> >>> > >>>>>>>>>>>> bus...@apache.org <javascript:;>
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> I updated all of our jobs to use the
> >>> > >> updated
> >>> > >>>>> JDK
> >>> > >>>>>>>>>> versions
> >>> > >>>>>>>>>>>>> from
> >>> > >>>>>>>>>>>>>>>>> infra.
> >>> > >>>>>>>>>>>>>>>>>>>>>> These have spaces in the names, and those
> >>> > >>>>> names
> >>> > >>>>>>> end
> >>> > >>>>>>>>> up
> >>> > >>>>>>>>>> in
> >>> > >>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>>>>>>> workspace path, so try to keep an eye
> >>> > >> out.
> >>> > >>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> On Mon, Aug 8, 2016 at 10:42 AM, Sean
> >>> > >>>> Busbey <
> >>> > >>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>> running in docker is the default now.
> >>> > >>>>> relying
> >>> > >>>>>>> on
> >>> > >>>>>>>>> the
> >>> > >>>>>>>>>>>>> default
> >>> > >>>>>>>>>>>>>>>>> docker
> >>> > >>>>>>>>>>>>>>>>>>>>>>> image that comes with Yetus means that
> >>> > >> our
> >>> > >>>>>>> protoc
> >>> > >>>>>>>>>>> checks
> >>> > >>>>>>>>>>>>> are
> >>> > >>>>>>>>>>>>>>>>>>>>>>> failing[1].
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>> [1]: https://issues.apache.org/
> >>> > >>>>>>>>>> jira/browse/HBASE-16373
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>> On Sat, Aug 6, 2016 at 5:03 PM, Sean
> >>> > >>>> Busbey
> >>> > >>>>> <
> >>> > >>>>>>>>>>>>>>>> bus...@apache.org <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi folks!
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>> this morning I merged the patch that
> >>> > >>>>> updates
> >>> > >>>>>>> us
> >>> > >>>>>>>> to
> >>> > >>>>>>>>>>> Yetus
> >>> > >>>>>>>>>>>>>>>>> 0.3.0[1]
> >>> > >>>>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>>>>> updated the precommit job appropriately.
> >>> > >> I
> >>> > >>>>> also
> >>> > >>>>>>>>> changed
> >>> > >>>>>>>>>>> it
> >>> > >>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>> use
> >>> > >>>>>>>>>>>>>>>>> one
> >>> > >>>>>>>>>>>>>>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>>>>>> the Java versions post the puppet
> >>> > >> changes to
> >>> > >>>>> asf
> >>> > >>>>>>>>> build.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>> The last three builds look normal
> >>> > >> (#2975
> >>> > >>>> -
> >>> > >>>>>>>> #2977).
> >>> > >>>>>>>>>> I'm
> >>> > >>>>>>>>>>>>> gonna
> >>> > >>>>>>>>>>>>>>>> try
> >>> > >>>>>>>>>>>>>>>>>>>>>> running things in docker next. I'll email
> >>> > >>>>> again
> >>> > >>>>>>>> when
> >>> > >>>>>>>>> I
> >>> > >>>>>>>>>>> make
> >>> > >>>>>>>>>>>>> it
> >>> > >>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>> default.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>> [1]: https://issues.apache.org/
> >>> > >>>>>>>>>>> jira/browse/HBASE-15882
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>> On 2016-06-16 10:43 (-0500), Sean
> >>> > >> Busbey
> >>> > >>>> <
> >>> > >>>>>>>>>>>>> bus...@apache.org <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> FYI, today our precommit jobs started
> >>> > >>>>> failing
> >>> > >>>>>>>>>> because
> >>> > >>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>> chosen
> >>> > >>>>>>>>>>>>>>>>>>> jdk
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> (1.7.0.79) disappeared (mentioned on
> >>> > >>>>>>>>> HBASE-16032).
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> Initially we were doing something
> >>> > >> wrong,
> >>> > >>>>>>> namely
> >>> > >>>>>>>>>>>> directly
> >>> > >>>>>>>>>>>>>>>>>>> referencing
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> the jenkins build tools area without
> >>> > >>>>> telling
> >>> > >>>>>>>>>> jenkins
> >>> > >>>>>>>>>>> to
> >>> > >>>>>>>>>>>>> give
> >>> > >>>>>>>>>>>>>>>>> us an
> >>> > >>>>>>>>>>>>>>>>>>>>> env
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> variable that stated where the jdk is
> >>> > >>>>>>> located.
> >>> > >>>>>>>>>>> However,
> >>> > >>>>>>>>>>>>>>>> after
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> attempting to switch to the
> >>> > >> appropriate
> >>> > >>>>>>> tooling
> >>> > >>>>>>>>>>>> variable
> >>> > >>>>>>>>>>>>> for
> >>> > >>>>>>>>>>>>>>>>> jdk
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> 1.7.0.79, I found that it didn't
> >>> > >> point
> >>> > >>>> to
> >>> > >>>>> a
> >>> > >>>>>>>> place
> >>> > >>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>> worked.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> I've now updated the job to rely on
> >>> > >> the
> >>> > >>>>>>> latest
> >>> > >>>>>>>>> 1.7
> >>> > >>>>>>>>>>> jdk,
> >>> > >>>>>>>>>>>>>>>> which
> >>> > >>>>>>>>>>>>>>>>> is
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> currently 1.7.0.80. I don't know how
> >>> > >>>> often
> >>> > >>>>>>>>> "latest"
> >>> > >>>>>>>>>>>>> updates.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> Personally, I think this is a sign
> >>> > >> that
> >>> > >>>> we
> >>> > >>>>>>> need
> >>> > >>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>> prioritize
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> HBASE-15882 so that we can switch
> >>> > >> back
> >>> > >>>> to
> >>> > >>>>>>> using
> >>> > >>>>>>>>>>>> Docker. I
> >>> > >>>>>>>>>>>>>>>> won't
> >>> > >>>>>>>>>>>>>>>>>>> have
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> time this week, so if anyone else
> >>> > >> does
> >>> > >>>>> please
> >>> > >>>>>>>>> pick
> >>> > >>>>>>>>>> up
> >>> > >>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>> ticket.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Mar 17, 2016 at 5:19 PM,
> >>> > >> Stack <
> >>> > >>>>>>>>>>>> st...@duboce.net <javascript:;>
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Sean.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 12:04 PM,
> >>> > >> Sean
> >>> > >>>>>>>> Busbey <
> >>> > >>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>
> >>> > >>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I updated the precommit job
> >>> > >>>> today
> >>> > >>>>> to
> >>> > >>>>>>>>>> specify
> >>> > >>>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>> only
> >>> > >>>>>>>>>>>>>>>>>>>>> compile
> >>> > >>>>>>>>>>>>>>>>>>>>>> time
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> checks should be done against jdks
> >>> > >>>>> other
> >>> > >>>>>>>> than
> >>> > >>>>>>>>>> the
> >>> > >>>>>>>>>>>>> primary
> >>> > >>>>>>>>>>>>>>>>> jdk7
> >>> > >>>>>>>>>>>>>>>>>>>>>> instance.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 8:43 PM,
> >>> > >> Sean
> >>> > >>>>>>> Busbey
> >>> > >>>>>>>> <
> >>> > >>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I tested things out, and while
> >>> > >>>>>>>> YETUS-297[1]
> >>> > >>>>>>>>> is
> >>> > >>>>>>>>>>>>> present
> >>> > >>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>>> default runs
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> all plugins that can do multiple
> >>> > >>>> jdks
> >>> > >>>>>>>>> against
> >>> > >>>>>>>>>>>> those
> >>> > >>>>>>>>>>>>>>>>> available
> >>> > >>>>>>>>>>>>>>>>>>>>>> (jdk7 and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> jdk8 in our case).
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> We can configure things to only
> >>> > >> do
> >>> > >>>> a
> >>> > >>>>>>>> single
> >>> > >>>>>>>>>> run
> >>> > >>>>>>>>>>> of
> >>> > >>>>>>>>>>>>> unit
> >>> > >>>>>>>>>>>>>>>>>>> tests.
> >>> > >>>>>>>>>>>>>>>>>>>>>> They'll be
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> against jdk7, since that is our
> >>> > >>>>> default
> >>> > >>>>>>>> jdk.
> >>> > >>>>>>>>>>> That
> >>> > >>>>>>>>>>>>> fine
> >>> > >>>>>>>>>>>>>>>> by
> >>> > >>>>>>>>>>>>>>>>>>>>>> everyone? It'll
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> save ~1.5 hours on any build
> >>> > >> that
> >>> > >>>>> hits
> >>> > >>>>>>>>>>>> hbase-server.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 1:22 PM,
> >>> > >>>>> Stack <
> >>> > >>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hurray!
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It looks like YETUS-96 is in
> >>> > >> there
> >>> > >>>>> and
> >>> > >>>>>>> we
> >>> > >>>>>>>>> are
> >>> > >>>>>>>>>>>> only
> >>> > >>>>>>>>>>>>>>>>> running
> >>> > >>>>>>>>>>>>>>>>>>> on
> >>> > >>>>>>>>>>>>>>>>>>>>>> jdk build
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> now, the default (but testing
> >>> > >>>>> compile
> >>> > >>>>>>>>> against
> >>> > >>>>>>>>>>>>>>>> both)....
> >>> > >>>>>>>>>>>>>>>>> Will
> >>> > >>>>>>>>>>>>>>>>>>>>>> keep an
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> eye.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Mar 7, 2016 at 10:27
> >>> > >> AM,
> >>> > >>>>> Sean
> >>> > >>>>>>>>> Busbey
> >>> > >>>>>>>>>> <
> >>> > >>>>>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I've just updated our
> >>> > >>>>> precommit
> >>> > >>>>>>>> jobs
> >>> > >>>>>>>>>> to
> >>> > >>>>>>>>>>>> use
> >>> > >>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>> 0.2.0
> >>> > >>>>>>>>>>>>>>>>>>>>>> release of
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yetus
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that came out today.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After keeping an eye out for
> >>> > >>>>>>>> strangeness
> >>> > >>>>>>>>>>> today
> >>> > >>>>>>>>>>>>> I'll
> >>> > >>>>>>>>>>>>>>>>> turn
> >>> > >>>>>>>>>>>>>>>>>>>>>> docker mode
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> back
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on by default tonight.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jan 13, 2016 at 10:14
> >>> > >>>> AM,
> >>> > >>>>>>> Sean
> >>> > >>>>>>>>>>> Busbey <
> >>> > >>>>>>>>>>>>>>>>>>>>>> bus...@apache.org <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I added a new
> >>> > >> parameter
> >>> > >>>> to
> >>> > >>>>> the
> >>> > >>>>>>>>>>> precommit
> >>> > >>>>>>>>>>>>> job:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * USE_YETUS_PRERELEASE -
> >>> > >>>> causes
> >>> > >>>>> us
> >>> > >>>>>>> to
> >>> > >>>>>>>>> use
> >>> > >>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>> HEAD of
> >>> > >>>>>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> apache/yetus
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repo rather than our chosen
> >>> > >>>>> release
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It defaults to inactive,
> >>> > >> but
> >>> > >>>>> can be
> >>> > >>>>>>>>> used
> >>> > >>>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>>>>>>> manually-triggered runs
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> test a solution to a
> >>> > >> problem
> >>> > >>>> in
> >>> > >>>>> the
> >>> > >>>>>>>>> yetus
> >>> > >>>>>>>>>>>>>>>> library. At
> >>> > >>>>>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>>> moment,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> using it to test a
> >>> > >> solution to
> >>> > >>>>>>>> default
> >>> > >>>>>>>>>>> module
> >>> > >>>>>>>>>>>>>>>>> ordering
> >>> > >>>>>>>>>>>>>>>>>>> as
> >>> > >>>>>>>>>>>>>>>>>>>>>> seen in
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HBASE-15075.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 8, 2016 at 7:58
> >>> > >>>> AM,
> >>> > >>>>>>> Sean
> >>> > >>>>>>>>>>> Busbey <
> >>> > >>>>>>>>>>>>>>>>>>>>>> bus...@cloudera.com <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, I just pushed
> >>> > >>>> HBASE-13525
> >>> > >>>>>>>>> (switch
> >>> > >>>>>>>>>> to
> >>> > >>>>>>>>>>>>> Apache
> >>> > >>>>>>>>>>>>>>>>> Yetus
> >>> > >>>>>>>>>>>>>>>>>>>>> for
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> precommit
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests)
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and updated our jenkins
> >>> > >>>>> precommit
> >>> > >>>>>>>>> build
> >>> > >>>>>>>>>>> to
> >>> > >>>>>>>>>>>>> use
> >>> > >>>>>>>>>>>>>>>> it.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jenkins job has some
> >>> > >>>>> explanation:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>>>>>>>>>>> view/PreCommit%20Builds/job/
> >>> > >>>>>>>>>>>>>>>>>>>>>> PreCommit-HBASE-Build/
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Release note from
> >>> > >>>> HBASE-13525
> >>> > >>>>>>> does
> >>> > >>>>>>>> as
> >>> > >>>>>>>>>>> well.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The old job will stick
> >>> > >>>> around
> >>> > >>>>>>> here
> >>> > >>>>>>>>> for
> >>> > >>>>>>>>>> a
> >>> > >>>>>>>>>>>>> couple
> >>> > >>>>>>>>>>>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>>> weeks,
> >>> > >>>>>>>>>>>>>>>>>>>>>> in case
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to refer back to it:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>>>>>>>>>>> view/PreCommit%20Builds/job/
> >>> > >>>>>>>>>>>>>>>>>>>>>> PreCommit-HBASE-Build-deprecated/
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If something looks awry,
> >>> > >>>>> please
> >>> > >>>>>>>> drop
> >>> > >>>>>>>>> a
> >>> > >>>>>>>>>>> note
> >>> > >>>>>>>>>>>>> on
> >>> > >>>>>>>>>>>>>>>>>>>>> HBASE-13525
> >>> > >>>>>>>>>>>>>>>>>>>>>> while
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remains
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> open (and make a new
> >>> > >> issue
> >>> > >>>>>>> after).
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Dec 2, 2015 at
> >>> > >> 3:22
> >>> > >>>>> PM,
> >>> > >>>>>>>>> Stack <
> >>> > >>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As part of my continuing
> >>> > >>>>>>> advocacy
> >>> > >>>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>>> builds.apache.org
> >>> > >>>>>>>>>>>>>>>>>>>>>> and that
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> their
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> results are now worthy
> >>> > >> of
> >>> > >>>> our
> >>> > >>>>>>>> trust
> >>> > >>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>> nurture,
> >>> > >>>>>>>>>>>>>>>>> here
> >>> > >>>>>>>>>>>>>>>>>>>>> are
> >>> > >>>>>>>>>>>>>>>>>>>>>> some
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> highlights
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the last few days
> >>> > >> of
> >>> > >>>>>>> builds:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + hadoopqa is now
> >>> > >> finding
> >>> > >>>>>>> zombies
> >>> > >>>>>>>>>> before
> >>> > >>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>> patch is
> >>> > >>>>>>>>>>>>>>>>>>>>>> committed.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HBASE-14888 showed "-1
> >>> > >> core
> >>> > >>>>>>> tests.
> >>> > >>>>>>>>> The
> >>> > >>>>>>>>>>>> patch
> >>> > >>>>>>>>>>>>>>>>> failed
> >>> > >>>>>>>>>>>>>>>>>>>>> these
> >>> > >>>>>>>>>>>>>>>>>>>>>> unit
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests:"
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> didn't have any failed
> >>> > >>>> tests
> >>> > >>>>>>>> listed
> >>> > >>>>>>>>>> (I'm
> >>> > >>>>>>>>>>>>>>>> trying to
> >>> > >>>>>>>>>>>>>>>>>>> see
> >>> > >>>>>>>>>>>>>>>>>>>>> if
> >>> > >>>>>>>>>>>>>>>>>>>>>> I can
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> do
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> anything
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about this...). Running
> >>> > >> our
> >>> > >>>>>>> little
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> ./dev-tools/findHangingTests.py
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> against
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the consoleText, it
> >>> > >> showed
> >>> > >>>> a
> >>> > >>>>>>>> hanging
> >>> > >>>>>>>>>>> test.
> >>> > >>>>>>>>>>>>>>>> Running
> >>> > >>>>>>>>>>>>>>>>>>>>>> locally, I see
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> same
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hang. This is before the
> >>> > >>>>> patch
> >>> > >>>>>>>>> landed.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + Our branch runs are
> >>> > >> now
> >>> > >>>>> near
> >>> > >>>>>>>>> totally
> >>> > >>>>>>>>>>>>> zombie
> >>> > >>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>> flakey
> >>> > >>>>>>>>>>>>>>>>>>>>>> free --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work to do -- but a
> >>> > >> recent
> >>> > >>>>> patch
> >>> > >>>>>>>>> that
> >>> > >>>>>>>>>>>> seemed
> >>> > >>>>>>>>>>>>>>>>> harmless
> >>> > >>>>>>>>>>>>>>>>>>>>> was
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> causing a
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reliable flake fail in
> >>> > >> the
> >>> > >>>>>>>> backport
> >>> > >>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>> branch-1*
> >>> > >>>>>>>>>>>>>>>>>>>>>> confirmed by
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> local
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The flakeyness was
> >>> > >> plain to
> >>> > >>>>> see
> >>> > >>>>>>> up
> >>> > >>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>>>> builds.apache.org
> >>> > >>>>>>>>>>>>>>>>>>>>> .
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> + In the last few days
> >>> > >> I've
> >>> > >>>>>>>>> committed
> >>> > >>>>>>>>>> a
> >>> > >>>>>>>>>>>>> patch
> >>> > >>>>>>>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>>>>>>>> included
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> javadoc
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warnings even though
> >>> > >>>> hadoopqa
> >>> > >>>>>>> said
> >>> > >>>>>>>>> the
> >>> > >>>>>>>>>>>> patch
> >>> > >>>>>>>>>>>>>>>>>>> introduced
> >>> > >>>>>>>>>>>>>>>>>>>>>> javadoc
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> issues
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (I
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> missed it). This messed
> >>> > >> up
> >>> > >>>>> life
> >>> > >>>>>>>> for
> >>> > >>>>>>>>>>> folks
> >>> > >>>>>>>>>>>>>>>>>>> subsequently
> >>> > >>>>>>>>>>>>>>>>>>>>> as
> >>> > >>>>>>>>>>>>>>>>>>>>>> their
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> patches
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reported javadoc
> >>> > >> issues....
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In short, I suggest that
> >>> > >>>>>>>>>>>> builds.apache.org
> >>> > >>>>>>>>>>>>> is
> >>> > >>>>>>>>>>>>>>>>> worth
> >>> > >>>>>>>>>>>>>>>>>>>>>> keeping an
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> eye
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sure you get a clean
> >>> > >> build
> >>> > >>>>> out
> >>> > >>>>>>> of
> >>> > >>>>>>>>>>> hadoopqa
> >>> > >>>>>>>>>>>>>>>> before
> >>> > >>>>>>>>>>>>>>>>>>>>>> committing
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> anything,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lets all work together
> >>> > >> to
> >>> > >>>> try
> >>> > >>>>>>> and
> >>> > >>>>>>>>> keep
> >>> > >>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>> builds
> >>> > >>>>>>>>>>>>>>>>>>> blue:
> >>> > >>>>>>>>>>>>>>>>>>>>>> it'll
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> save
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> us
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work in the long run.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Nov 4, 2014 at
> >>> > >> 9:38
> >>> > >>>>> AM,
> >>> > >>>>>>>>> Stack
> >>> > >>>>>>>>>> <
> >>> > >>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>
> >>> > >>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Branch-1 and master
> >>> > >> have
> >>> > >>>>>>>>> stabilized
> >>> > >>>>>>>>>>> and
> >>> > >>>>>>>>>>>>> now
> >>> > >>>>>>>>>>>>>>>> run
> >>> > >>>>>>>>>>>>>>>>>>> mostly
> >>> > >>>>>>>>>>>>>>>>>>>>>> blue
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> (give or
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the odd failure)
> >>> > >> [1][2].
> >>> > >>>>>>> Having
> >>> > >>>>>>>> a
> >>> > >>>>>>>>>>> mostly
> >>> > >>>>>>>>>>>>> blue
> >>> > >>>>>>>>>>>>>>>>>>> branch-1
> >>> > >>>>>>>>>>>>>>>>>>>>>> has
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> helped us
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> identify at least one
> >>> > >>>>>>>>> destabilizing
> >>> > >>>>>>>>>>>>> commit in
> >>> > >>>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>> last
> >>> > >>>>>>>>>>>>>>>>>>>>>> few
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> days,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maybe
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two;
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is as it should
> >>> > >> be
> >>> > >>>>>>> (smile).
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lets keep our builds
> >>> > >>>> blue.
> >>> > >>>>> If
> >>> > >>>>>>>> you
> >>> > >>>>>>>>>>>> commit a
> >>> > >>>>>>>>>>>>>>>>> patch,
> >>> > >>>>>>>>>>>>>>>>>>> make
> >>> > >>>>>>>>>>>>>>>>>>>>>> sure
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> subsequent
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> builds stay blue. You
> >>> > >> can
> >>> > >>>>>>>>> subscribe
> >>> > >>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>>>> bui...@hbase.apache.org <javascript:;>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> notice of failures if
> >>> > >> not
> >>> > >>>>>>>> already
> >>> > >>>>>>>>>>>>> subscribed.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/
> >>> > >>>>>>>>>>>>> view/H-L/view/HBase/job/HBase-
> >>> > >>>>>>>>>>>>>>>>> 1.0/
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://builds.apache.org/view
> >>> > >>>>>>>>>>>>>>>> /H-L/view/HBase/job/HBase-
> >>> > >>>>>>>>>>>>>>>>>>> TRUNK/
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 13, 2014
> >>> > >> at
> >>> > >>>>> 4:41
> >>> > >>>>>>> PM,
> >>> > >>>>>>>>>>> Stack <
> >>> > >>>>>>>>>>>>>>>>>>>>>> st...@duboce.net <javascript:;>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A few notes on
> >>> > >> testing.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Too long to read,
> >>> > >> infra
> >>> > >>>> is
> >>> > >>>>>>> more
> >>> > >>>>>>>>>>> capable
> >>> > >>>>>>>>>>>>> now
> >>> > >>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>> after
> >>> > >>>>>>>>>>>>>>>>>>>>>> some
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> work, we
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seeing branch-1 and
> >>> > >>>> trunk
> >>> > >>>>>>>> mostly
> >>> > >>>>>>>>>>>> running
> >>> > >>>>>>>>>>>>>>>> blue.
> >>> > >>>>>>>>>>>>>>>>>>> Lets
> >>> > >>>>>>>>>>>>>>>>>>>>>> try and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> keep it
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> way going forward.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Infra has new,
> >>> > >>>> more
> >>> > >>>>>>>>> capable
> >>> > >>>>>>>>>>>>> hardware.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A recent spurt of
> >>> > >> test
> >>> > >>>>> fixing
> >>> > >>>>>>>>>>> combined
> >>> > >>>>>>>>>>>>> with
> >>> > >>>>>>>>>>>>>>>>> more
> >>> > >>>>>>>>>>>>>>>>>>>>>> capable
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> hardware
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to have gotten us to
> >>> > >> a
> >>> > >>>> new
> >>> > >>>>>>>> place;
> >>> > >>>>>>>>>>> tests
> >>> > >>>>>>>>>>>>> are
> >>> > >>>>>>>>>>>>>>>>> mostly
> >>> > >>>>>>>>>>>>>>>>>>>>>> passing now
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> branch-1
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and master. Lets try
> >>> > >>>> and
> >>> > >>>>>>> keep
> >>> > >>>>>>>> it
> >>> > >>>>>>>>>>> this
> >>> > >>>>>>>>>>>>> way
> >>> > >>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>> start
> >>> > >>>>>>>>>>>>>>>>>>>>>> to trust
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> test
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runs
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> again. Just a few
> >>> > >>>> flakies
> >>> > >>>>>>>>> remain.
> >>> > >>>>>>>>>>>> Lets
> >>> > >>>>>>>>>>>>> try
> >>> > >>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>> nail
> >>> > >>>>>>>>>>>>>>>>>>>>>> them.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Our tests now run in
> >>> > >>>>> parallel
> >>> > >>>>>>>>> with
> >>> > >>>>>>>>>>>> other
> >>> > >>>>>>>>>>>>>>>> test
> >>> > >>>>>>>>>>>>>>>>>>> suites
> >>> > >>>>>>>>>>>>>>>>>>>>>> where
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ran alone. You can
> >>> > >> see
> >>> > >>>>> this
> >>> > >>>>>>>>>> sometimes
> >>> > >>>>>>>>>>>>> when
> >>> > >>>>>>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>>>> zombie
> >>> > >>>>>>>>>>>>>>>>>>>>>> detector
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reports
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests from another
> >>> > >>>> project
> >>> > >>>>>>>>>> altogether
> >>> > >>>>>>>>>>>> as
> >>> > >>>>>>>>>>>>>>>>> lingerers
> >>> > >>>>>>>>>>>>>>>>>>>>> (To
> >>> > >>>>>>>>>>>>>>>>>>>>>> be
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixed).
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Some
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our tests are failing
> >>> > >>>>>>> because a
> >>> > >>>>>>>>>>>>> concurrent
> >>> > >>>>>>>>>>>>>>>>> hbase
> >>> > >>>>>>>>>>>>>>>>>>> run
> >>> > >>>>>>>>>>>>>>>>>>>>> is
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> undoing
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> classes
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data from under it.
> >>> > >>>> Also,
> >>> > >>>>>>> lets
> >>> > >>>>>>>>> fix.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Our tests are
> >>> > >> brittle.
> >>> > >>>> It
> >>> > >>>>>>> takes
> >>> > >>>>>>>>>>>> 75minutes
> >>> > >>>>>>>>>>>>>>>> for
> >>> > >>>>>>>>>>>>>>>>>>> them to
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> complete.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Many
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> heavy-duty
> >>> > >> integration
> >>> > >>>>> tests
> >>> > >>>>>>>>>> starting
> >>> > >>>>>>>>>>>> up
> >>> > >>>>>>>>>>>>>>>>> multiple
> >>> > >>>>>>>>>>>>>>>>>>>>>> clusters and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mapreduce
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all in the one JVM.
> >>> > >> It
> >>> > >>>> is
> >>> > >>>>> a
> >>> > >>>>>>>>> miracle
> >>> > >>>>>>>>>>>> they
> >>> > >>>>>>>>>>>>>>>> pass
> >>> > >>>>>>>>>>>>>>>>> at
> >>> > >>>>>>>>>>>>>>>>>>> all.
> >>> > >>>>>>>>>>>>>>>>>>>>>> Usually
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integration
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tests have been cast
> >>> > >> as
> >>> > >>>>> unit
> >>> > >>>>>>>>> tests
> >>> > >>>>>>>>>>>>> because
> >>> > >>>>>>>>>>>>>>>>> there
> >>> > >>>>>>>>>>>>>>>>>>> was
> >>> > >>>>>>>>>>>>>>>>>>>>>> no where
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> else
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to get an airing. We
> >>> > >>>> have
> >>> > >>>>>>> the
> >>> > >>>>>>>>>>> hbase-it
> >>> > >>>>>>>>>>>>>>>> suite
> >>> > >>>>>>>>>>>>>>>>> now
> >>> > >>>>>>>>>>>>>>>>>>>>>> which would
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apt
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> place but until these
> >>> > >>>> are
> >>> > >>>>> run
> >>> > >>>>>>>> on
> >>> > >>>>>>>>> a
> >>> > >>>>>>>>>>>>> regular
> >>> > >>>>>>>>>>>>>>>>> basis
> >>> > >>>>>>>>>>>>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>>>>>>> public for
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the fat integration
> >>> > >>>> tests
> >>> > >>>>>>>>> disguised
> >>> > >>>>>>>>>>> as
> >>> > >>>>>>>>>>>>> unit
> >>> > >>>>>>>>>>>>>>>>> tests
> >>> > >>>>>>>>>>>>>>>>>>>>> will
> >>> > >>>>>>>>>>>>>>>>>>>>>> remain.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> A
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> review of
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our current unit
> >>> > >> tests
> >>> > >>>>>>> weeding
> >>> > >>>>>>>>> the
> >>> > >>>>>>>>>>> old
> >>> > >>>>>>>>>>>>> cruft
> >>> > >>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>>> no longer
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> relevant
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> duplicates would be a
> >>> > >>>> nice
> >>> > >>>>>>>>>>> undertaking
> >>> > >>>>>>>>>>>> if
> >>> > >>>>>>>>>>>>>>>>> someone
> >>> > >>>>>>>>>>>>>>>>>>> is
> >>> > >>>>>>>>>>>>>>>>>>>>>> looking
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> contribute.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alex Newman has been
> >>> > >>>>> working
> >>> > >>>>>>> on
> >>> > >>>>>>>>>>> making
> >>> > >>>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>> tests
> >>> > >>>>>>>>>>>>>>>>>>> work
> >>> > >>>>>>>>>>>>>>>>>>>>>> up on
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> travis
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> circle-ci. That'll
> >>> > >> be
> >>> > >>>>> sweet
> >>> > >>>>>>>> when
> >>> > >>>>>>>>>> it
> >>> > >>>>>>>>>>>> goes
> >>> > >>>>>>>>>>>>>>>>>>> end-to-end.
> >>> > >>>>>>>>>>>>>>>>>>>>>> He also
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> added
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some "type"
> >>> > >>>>> categorizations
> >>> > >>>>>>> --
> >>> > >>>>>>>>>>> client,
> >>> > >>>>>>>>>>>>>>>> filter,
> >>> > >>>>>>>>>>>>>>>>>>>>>> mapreduce --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> alongside
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> our
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> old "sizing"
> >>> > >>>>> categorizations
> >>> > >>>>>>> of
> >>> > >>>>>>>>>>>>>>>>>>> small/medium/large.
> >>> > >>>>>>>>>>>>>>>>>>>>>> His
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> thinking
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we can run these
> >>> > >>>>>>>> categorizations
> >>> > >>>>>>>>> in
> >>> > >>>>>>>>>>>>> parallel
> >>> > >>>>>>>>>>>>>>>>> so we
> >>> > >>>>>>>>>>>>>>>>>>>>>> could run
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> total
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suite in about the
> >>> > >> time
> >>> > >>>> of
> >>> > >>>>>>> the
> >>> > >>>>>>>>>>> longest
> >>> > >>>>>>>>>>>>> test,
> >>> > >>>>>>>>>>>>>>>>> say
> >>> > >>>>>>>>>>>>>>>>>>>>>> 20-30minutes?
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> even change Apache to
> >>> > >>>> run
> >>> > >>>>>>> them
> >>> > >>>>>>>>> this
> >>> > >>>>>>>>>>>> way.
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI,
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> St.Ack
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sean
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>>> Sean
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>> Thanks,
> >>> > >>>>>>>>>>>>>>>> Michael Antonov
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>> -- Appy
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>> -- Appy
> >>> > >>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>>
> >>> > >>>>>>>>>>>>> --
> >>> > >>>>>>>>>>>>> busbey
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>>
> >>> > >>>>>>>>>>>> --
> >>> > >>>>>>>>>>>> Thanks,
> >>> > >>>>>>>>>>>> Michael Antonov
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>> --
> >>> > >>>>>>>>>>>
> >>> > >>>>>>>>>>> -- Appy
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>
> >>> > >>>>>>>>>
> >>> > >>>>>>>>> --
> >>> > >>>>>>>>>
> >>> > >>>>>>>>> -- Appy
> >>> > >>>>>>>>
> >>> > >>>>>>>>
> >>> > >>>>>>>> --
> >>> > >>>>>>>> -Dima
> >>> > >>>>>>
> >>> > >>>>>>
> >>> > >>>>>>
> >>> > >>>>>> --
> >>> > >>>>>>
> >>> > >>>>>> -- Appy
> >>> > >>>>>
> >>> > >>>>>
> >>> > >>>>>
> >>> > >>>>> --
> >>> > >>>>> busbey
> >>> > >>>
> >>> > >>>
> >>> > >>>
> >>> > >>> --
> >>> > >>>
> >>> > >>> -- Appy
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >> busbey
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best regards,
> >>> > >
> >>> > > - Andy
> >>> > >
> >>> > > Problems worthy of attack prove their worth by hitting back. - Piet
> >>> Hein
> >>> > > (via Tom White)
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>>
> >>> -- Appy
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
>
--
Thanks,
Michael Antonov
