Clues on how to figure root cause of:

[ERROR] Error invoking method 'get(java.lang.Integer)' in
java.util.ArrayList at META-INF/LICENSE.vm[line 1627, column 22]


.... comes of the initial findbugs run when it goes into assembly. See here
in Trunk matrix:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/jdk=latest1.7,label=Hadoop/430/console

Then the checkstyle run fails with this in another 1.7 trunk matrix run:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.16:checkstyle
(default-cli) on project hbase-external-blockcache: An error has
occurred in Checkstyle report generation. Failed during checkstyle
execution: Unable to process suppressions file location:
hbase/checkstyle-suppressions.xml: Cannot create file-based
resource:invalid block type -> [Help 1]


Thanks,
St.Ack



On Tue, Nov 3, 2015 at 10:31 AM, Stack <st...@duboce.net> wrote:

> Maybe it got reenabled. I've disabled it now. Thanks Sean.
> St.Ack
>
> On Tue, Nov 3, 2015 at 9:50 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> I thought HBase_TRUNK was already disabled. if I missed it, disabling
>> should be sufficient. There are a few older non-matrix (0.98 I think?)
>> that we can delete once we're confident in the matrix version
>>
>> On Tue, Nov 3, 2015 at 8:44 AM, Stack <st...@duboce.net> wrote:
>> > Stuff seems to be basically working.  I just committed a fix for this
>> > message so it should be gone now:
>> >
>> > grep: unknown devices method
>> > grep: write error: Broken pipe
>> >
>> >
>> > Hopefully we've seen last of the errant surefire kills.
>> >
>> > We have an HBASE_TRUNK build and we have an HBASE_TRUNK_MATRIX. We
>> should
>> > get rid of the former now?
>> >
>> > St.Ack
>> >
>> > On Mon, Nov 2, 2015 at 11:03 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> I just did an edit of all of our jenkins configuration. I changed the
>> >> post-build zombie section. The surefire-killer was hanging out here it
>> >> seems (For detail, see HBASE-14589 Looking for the surefire-killer;
>> builds
>> >> being killed...). I may have messed up builds. Will fix in morning if
>> so
>> >> (currently builds seem a bit backed up so won't wait around).
>> >>
>> >> St.Ack
>> >>
>> >> On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <bus...@cloudera.com>
>> wrote:
>> >>
>> >>> Hi Folks,
>> >>>
>> >>> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS]
>> >>> reducing our load on builds.a.o' on reasoning or if you have concerns.
>> >>>
>> >>> Our project job view now has a brief summary of what's going on:
>> >>>
>> >>> https://builds.apache.org/view/H-L/view/HBase/
>> >>>
>> >>> * Where we had matrix builds, I confirmed they're doing the same thing
>> >>> (in at least one test-configuration) as the non-matrix version
>> >>> * I disabled jobs that were duplicating work done in matrix jobs, or
>> >>> that only worked on a tag
>> >>> * Made sure all non-disabled 0.98+ jobs are set to notify jira +
>> >>> builds mailing list
>> >>> * I updated pre-1.1 jobs to look for changes 1/day
>> >>> * I updated 1.1 jobs to look for changes every 8 hours
>> >>> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours
>> >>>
>> >>> There's still some non-pressing clean up work to do, so I'll file a
>> >>> jira later today just to make sure it's all recorded somewhere.
>> >>>
>> >>> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <bus...@cloudera.com>
>> >>> wrote:
>> >>> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2
>> >>> builds to
>> >>> > use branch-1.2. (ref HBASE-13912)
>> >>> >
>> >>> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <bus...@cloudera.com>
>> >>> wrote:
>> >>> >>
>> >>> >> I've add a new build test to run the ITs under sunny day conditions
>> >>> using
>> >>> >> minicluster for both java 7 and java 8 on the 1.2 line.
>> >>> >>
>> >>> >> https://builds.apache.org/job/HBase-1.2-IT/
>> >>> >>
>> >>> >> I haven't turned on notifications yet, because BulkIngest and
>> >>> TestIngest
>> >>> >> are read. details on HBASE-13750
>> >>> >>
>> >>> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <bus...@cloudera.com>
>> >>> wrote:
>> >>> >>>
>> >>> >>> sigh. that should have been "is now a matrix build".
>> >>> >>>
>> >>> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <bus...@cloudera.com
>> >
>> >>> wrote:
>> >>> >>>>
>> >>> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and
>> JDK8
>> >>> in
>> >>> >>>> parallel.
>> >>> >>>>
>> >>> >>>> I saved the old job for now and left it disabled[2].
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> [1]:
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
>> >>> >>>> [2]:
>> >>> >>>>
>> >>>
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
>> >>> >>>>
>> >>> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <
>> bus...@cloudera.com>
>> >>> >>>> wrote:
>> >>> >>>>>
>> >>> >>>>> Sorry for the noise. I also updated the checkstyle step on
>> >>> HBase-TRUNK
>> >>> >>>>>
>> >>> >>>>> from
>> >>> >>>>>     mvn checkstyle:checkstyle
>> >>> >>>>> to
>> >>> >>>>>     mvn -DskipTests package checkstyle:checkstyle
>> >>> >>>>>
>> >>> >>>>> to deal with the same issue. Looks all clear now.
>> >>> >>>>>
>> >>> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <
>> bus...@cloudera.com>
>> >>> >>>>> wrote:
>> >>> >>>>>>
>> >>> >>>>>> Updated the following builds:
>> >>> >>>>>>
>> >>> >>>>>> * HBase-TRUNK
>> >>> >>>>>>
>> >>> >>>>>> moved from
>> >>> >>>>>>
>> >>> >>>>>>     mvn clean compile findbugs:findbugs
>> >>> >>>>>>
>> >>> >>>>>> to
>> >>> >>>>>>
>> >>> >>>>>>    mvn clean -DskipTests package findbugs:findbugs
>> >>> >>>>>>
>> >>> >>>>>> To work around the known issue where we can't do a bootstrap
>> >>> compile
>> >>> >>>>>> without previous install or remote SNAPSHOT artifacts.
>> (recently
>> >>> triggered
>> >>> >>>>>> by the addition of the procedure module on master)
>> >>> >>>>>>
>> >>> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <palomino...@gmail.com>
>> wrote:
>> >>> >>>>>>>
>> >>> >>>>>>> Make findbugs and checkstyle plugins always run.
>> >>> >>>>>>> The default behavior only publishes result for stable builds,
>> but
>> >>> >>>>>>> master is
>> >>> >>>>>>> not always stable and sometimes we introduce new warnings in
>> >>> unstable
>> >>> >>>>>>> builds(see builds 6310-6314).
>> >>> >>>>>>> And findbugs and checkstyle will not fail unless we abort the
>> >>> >>>>>>> building
>> >>> >>>>>>> process I think, so it is safe to turn it on every time.
>> >>> >>>>>>>
>> >>> >>>>>>> Thanks.
>> >>> >>>>>>>
>> >>> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>> >>> >>>>>>>
>> >>> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report.
>> >>> >>>>>>> >
>> >>> >>>>>>> > Cheers
>> >>> >>>>>>> >
>> >>> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <palomino...@gmail.com>
>> >>> wrote:
>> >>> >>>>>>> >
>> >>> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show
>> >>> >>>>>>> > > findbugs,
>> >>> >>>>>>> > > checkstyle and jacoco coverage report.
>> >>> >>>>>>> > > The config has been tested on
>> >>> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for
>> nearly
>> >>> 30
>> >>> >>>>>>> > > times.
>> >>> >>>>>>> > Not
>> >>> >>>>>>> > > much different from HBase-TRUNK unless it runs ~30%
>> slower(the
>> >>> >>>>>>> > > overhead
>> >>> >>>>>>> > of
>> >>> >>>>>>> > > collecting information for code coverage).
>> >>> >>>>>>> > > Thanks.
>> >>> >>>>>>> > >
>> >>> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <
>> apurt...@apache.org
>> >>> >:
>> >>> >>>>>>> > >
>> >>> >>>>>>> > > > +1, thanks a lot for improving our build hygiene.
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar
>> >>> >>>>>>> > > > <enis....@gmail.com>
>> >>> >>>>>>> > > wrote:
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > > > Thanks Sean. This is great.
>> >>> >>>>>>> > > > >
>> >>> >>>>>>> > > > > Enis
>> >>> >>>>>>> > > > >
>> >>> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey
>> >>> >>>>>>> > > > > <bus...@cloudera.com>
>> >>> >>>>>>> > > > wrote:
>> >>> >>>>>>> > > > >
>> >>> >>>>>>> > > > > > FYI I just finished chasing down the breakage for
>> mvn
>> >>> site
>> >>> >>>>>>> > > > > > on all
>> >>> >>>>>>> > > patch
>> >>> >>>>>>> > > > > > builds.
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > HBASE-13191 consolidates the few places in the
>> >>> test-patch
>> >>> >>>>>>> > > > > > code
>> >>> >>>>>>> > where
>> >>> >>>>>>> > > we
>> >>> >>>>>>> > > > > > hard coded MAVEN_OPTS.
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > If you look at the PreCommit job now, we use the
>> "set
>> >>> >>>>>>> > > > > > environment
>> >>> >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then
>> everything
>> >>> >>>>>>> > > > > > else
>> >>> >>>>>>> > respects
>> >>> >>>>>>> > > > > that
>> >>> >>>>>>> > > > > > setting.
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > I set the initial value to be a combination of the
>> >>> memory
>> >>> >>>>>>> > limitations
>> >>> >>>>>>> > > > > we've
>> >>> >>>>>>> > > > > > been actually running with (the ~6G was getting
>> ignored)
>> >>> >>>>>>> > > > > > and the
>> >>> >>>>>>> > > > permgen
>> >>> >>>>>>> > > > > > needed for site.
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData
>> >>> -XX:MaxPermSize=256m
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack <
>> >>> st...@duboce.net>
>> >>> >>>>>>> > > > > > wrote:
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same
>> jvm as
>> >>> >>>>>>> > > > > > > patch-build
>> >>> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the
>> MAVEN_OPTS
>> >>> to
>> >>> >>>>>>> > > > > > > be the
>> >>> >>>>>>> > > same
>> >>> >>>>>>> > > > as
>> >>> >>>>>>> > > > > > > those of trunk build too, setting
>> >>> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"... it
>> >>> >>>>>>> > had
>> >>> >>>>>>> > > > been
>> >>> >>>>>>> > > > > > > 3000.
>> >>> >>>>>>> > > > > > >
>> >>> >>>>>>> > > > > > > Yours,
>> >>> >>>>>>> > > > > > > St.Ack
>> >>> >>>>>>> > > > > > >
>> >>> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack <
>> >>> st...@duboce.net>
>> >>> >>>>>>> > > > > > > wrote:
>> >>> >>>>>>> > > > > > >
>> >>> >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100
>> builds
>> >>> and
>> >>> >>>>>>> > > > > > > > or last
>> >>> >>>>>>> > 7
>> >>> >>>>>>> > > > > days,
>> >>> >>>>>>> > > > > > > > whichever comes first.
>> >>> >>>>>>> > > > > > > > St.Ack
>> >>> >>>>>>> > > > > > > >
>> >>> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack
>> >>> >>>>>>> > > > > > > > <st...@duboce.net>
>> >>> >>>>>>> > wrote:
>> >>> >>>>>>> > > > > > > >
>> >>> >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run
>> >>> mostly
>> >>> >>>>>>> > > > > > > >> blue
>> >>> >>>>>>> > > (give
>> >>> >>>>>>> > > > or
>> >>> >>>>>>> > > > > > > take
>> >>> >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue
>> >>> branch-1
>> >>> >>>>>>> > > > > > > >> has
>> >>> >>>>>>> > > helped
>> >>> >>>>>>> > > > us
>> >>> >>>>>>> > > > > > > >> identify at least one destabilizing commit in
>> the
>> >>> last
>> >>> >>>>>>> > > > > > > >> few
>> >>> >>>>>>> > days,
>> >>> >>>>>>> > > > > maybe
>> >>> >>>>>>> > > > > > > two;
>> >>> >>>>>>> > > > > > > >> this is as it should be (smile).
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a
>> patch,
>> >>> make
>> >>> >>>>>>> > > > > > > >> sure
>> >>> >>>>>>> > > > > subsequent
>> >>> >>>>>>> > > > > > > >> builds stay blue. You can subscribe to
>> >>> >>>>>>> > bui...@hbase.apache.org
>> >>> >>>>>>> > > to
>> >>> >>>>>>> > > > > get
>> >>> >>>>>>> > > > > > > >> notice of failures if not already subscribed.
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >> Thanks,
>> >>> >>>>>>> > > > > > > >> St.Ack
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >> 1.
>> >>> >>>>>>> >
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
>> >>> >>>>>>> > > > > > > >> 2.
>> >>> >>>>>>> > >
>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack
>> >>> >>>>>>> > > > > > > >> <st...@duboce.net>
>> >>> >>>>>>> > > wrote:
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >>> A few notes on testing.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now
>> and
>> >>> after
>> >>> >>>>>>> > > > > > > >>> some
>> >>> >>>>>>> > > work,
>> >>> >>>>>>> > > > we
>> >>> >>>>>>> > > > > > are
>> >>> >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue.
>> >>> Lets
>> >>> >>>>>>> > > > > > > >>> try and
>> >>> >>>>>>> > > keep
>> >>> >>>>>>> > > > it
>> >>> >>>>>>> > > > > > > this
>> >>> >>>>>>> > > > > > > >>> way going forward.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with
>> more
>> >>> >>>>>>> > > > > > > >>> capable
>> >>> >>>>>>> > > hardware
>> >>> >>>>>>> > > > > > seems
>> >>> >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are
>> mostly
>> >>> >>>>>>> > > > > > > >>> passing
>> >>> >>>>>>> > now
>> >>> >>>>>>> > > on
>> >>> >>>>>>> > > > > > > branch-1
>> >>> >>>>>>> > > > > > > >>> and master.  Lets try and keep it this way and
>> >>> start
>> >>> >>>>>>> > > > > > > >>> to trust
>> >>> >>>>>>> > > our
>> >>> >>>>>>> > > > > > test
>> >>> >>>>>>> > > > > > > runs
>> >>> >>>>>>> > > > > > > >>> again.  Just a few flakies remain.  Lets try
>> and
>> >>> nail
>> >>> >>>>>>> > > > > > > >>> them.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test
>> >>> suites
>> >>> >>>>>>> > > > > > > >>> where
>> >>> >>>>>>> > > > previous
>> >>> >>>>>>> > > > > > we
>> >>> >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our
>> >>> zombie
>> >>> >>>>>>> > detector
>> >>> >>>>>>> > > > > > reports
>> >>> >>>>>>> > > > > > > >>> tests from another project altogether as
>> lingerers
>> >>> >>>>>>> > > > > > > >>> (To be
>> >>> >>>>>>> > > fixed).
>> >>> >>>>>>> > > > > > > Some of
>> >>> >>>>>>> > > > > > > >>> our tests are failing because a concurrent
>> hbase
>> >>> run
>> >>> >>>>>>> > > > > > > >>> is
>> >>> >>>>>>> > undoing
>> >>> >>>>>>> > > > > > > classes and
>> >>> >>>>>>> > > > > > > >>> data from under it. Also, lets fix.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for
>> >>> them to
>> >>> >>>>>>> > complete.
>> >>> >>>>>>> > > > > Many
>> >>> >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up
>> >>> multiple
>> >>> >>>>>>> > clusters
>> >>> >>>>>>> > > > and
>> >>> >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle
>> they
>> >>> >>>>>>> > > > > > > >>> pass at
>> >>> >>>>>>> > all.
>> >>> >>>>>>> > > > > > > Usually
>> >>> >>>>>>> > > > > > > >>> integration tests have been cast as unit tests
>> >>> >>>>>>> > > > > > > >>> because there
>> >>> >>>>>>> > > was
>> >>> >>>>>>> > > > no
>> >>> >>>>>>> > > > > > > where
>> >>> >>>>>>> > > > > > > >>> else for them to get an airing.  We have the
>> >>> hbase-it
>> >>> >>>>>>> > > > > > > >>> suite
>> >>> >>>>>>> > now
>> >>> >>>>>>> > > > > which
>> >>> >>>>>>> > > > > > > would
>> >>> >>>>>>> > > > > > > >>> be a more apt place but until these are run
>> on a
>> >>> >>>>>>> > > > > > > >>> regular
>> >>> >>>>>>> > basis
>> >>> >>>>>>> > > in
>> >>> >>>>>>> > > > > > > public
>> >>> >>>>>>> > > > > > > >>> for all to see, the fat integration tests
>> >>> disguised
>> >>> >>>>>>> > > > > > > >>> as unit
>> >>> >>>>>>> > > tests
>> >>> >>>>>>> > > > > > will
>> >>> >>>>>>> > > > > > > >>> remain.  A review of our current unit tests
>> >>> weeding
>> >>> >>>>>>> > > > > > > >>> the old
>> >>> >>>>>>> > > cruft
>> >>> >>>>>>> > > > > and
>> >>> >>>>>>> > > > > > > the
>> >>> >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a
>> nice
>> >>> >>>>>>> > > > > > > >>> undertaking
>> >>> >>>>>>> > if
>> >>> >>>>>>> > > > > > > someone is
>> >>> >>>>>>> > > > > > > >>> looking to contribute.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> Alex Newman has been working on making our
>> tests
>> >>> work
>> >>> >>>>>>> > > > > > > >>> up on
>> >>> >>>>>>> > > > travis
>> >>> >>>>>>> > > > > > and
>> >>> >>>>>>> > > > > > > >>> circle-ci.  That'll be sweet when it goes
>> >>> end-to-end.
>> >>> >>>>>>> > > > > > > >>> He
>> >>> >>>>>>> > also
>> >>> >>>>>>> > > > > added
>> >>> >>>>>>> > > > > > in
>> >>> >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter,
>> >>> >>>>>>> > > > > > > >>> mapreduce --
>> >>> >>>>>>> > > > > alongside
>> >>> >>>>>>> > > > > > > our
>> >>> >>>>>>> > > > > > > >>> old "sizing" categorizations of
>> >>> small/medium/large.
>> >>> >>>>>>> > > > > > > >>> His
>> >>> >>>>>>> > > thinking
>> >>> >>>>>>> > > > > is
>> >>> >>>>>>> > > > > > > that
>> >>> >>>>>>> > > > > > > >>> we can run these categorizations in parallel
>> so we
>> >>> >>>>>>> > > > > > > >>> could run
>> >>> >>>>>>> > > the
>> >>> >>>>>>> > > > > > total
>> >>> >>>>>>> > > > > > > >>> suite in about the time of the longest test,
>> say
>> >>> >>>>>>> > 20-30minutes?
>> >>> >>>>>>> > > > We
>> >>> >>>>>>> > > > > > > could
>> >>> >>>>>>> > > > > > > >>> even change Apache to run them this way.
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>> FYI,
>> >>> >>>>>>> > > > > > > >>> St.Ack
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>>
>> >>> >>>>>>> > > > > > > >>
>> >>> >>>>>>> > > > > > > >
>> >>> >>>>>>> > > > > > >
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > > > --
>> >>> >>>>>>> > > > > > Sean
>> >>> >>>>>>> > > > > >
>> >>> >>>>>>> > > > >
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > > --
>> >>> >>>>>>> > > > Best regards,
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > >    - Andy
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > > > Problems worthy of attack prove their worth by hitting
>> >>> back. -
>> >>> >>>>>>> > > > Piet
>> >>> >>>>>>> > Hein
>> >>> >>>>>>> > > > (via Tom White)
>> >>> >>>>>>> > > >
>> >>> >>>>>>> > >
>> >>> >>>>>>> >
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>>
>> >>> >>>>>> --
>> >>> >>>>>> Sean
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> --
>> >>> >>>>> Sean
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> --
>> >>> >>>> Sean
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>>
>> >>> >>> --
>> >>> >>> Sean
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Sean
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Sean
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Sean
>> >>>
>> >>
>> >>
>>
>>
>>
>> --
>> Sean
>>
>
>

Reply via email to