Cool!!! First 0.99 is out!! It's the first stone for the road to 1.0...

Enis, any concern if I relay the announcement you will send yo the Apache
Announce mailing list?

JM
Le 2014-09-21 19:45, "Enis Söztutar" <[email protected]> a écrit :

> The vote passed with 4 +1s. Although there are known problems in perf, I
> will continue with the release flow since this is a non-production release.
> The reason we are doing this release is to encourage more testing and
> finding out issues.
>
> I will send out announcement after 24 hours. Thanks everyone who have
> tested and voted.
>
> Enis
>
> On Sun, Sep 21, 2014 at 5:27 AM, Jean-Marc Spaggiari <
> [email protected]> wrote:
>
> > 1) Confirmed. 2nd run gave similar results.
> > 2) Checked the logs and so far, nothing wrong. Looking at the
> > FilteredScanTest code now.
> >
> > Opened HBASE-12040 and HBASE-12042
> >
> > 2014-09-21 7:48 GMT-04:00 Jean-Marc Spaggiari <[email protected]>:
> >
> > > Sorry, was HTML formated.
> > > 0.98.3 => 0,25
> > > 0.98.4 => 0,25
> > > 0.98.5 => 67,34
> > > 0.98.6 => 67,49
> > > 0.99.0 => 0,63
> > >
> > > On my todo:
> > > 1) Get updated numbers from 0.99.0
> > > 2) Take a look at the test code to make sure it still test the same
> > thing.
> > > 3) Re-run 0.98.4 and 0.98.5
> > >
> > >
> > > JM
> > >
> > > 2014-09-20 23:01 GMT-04:00 Ted Yu <[email protected]>:
> > >
> > > J-M:
> > >> Can you reformat the numbers below - it is hard to read.
> > >> Maybe put them in multiple rows.
> > >>
> > >> On Sat, Sep 20, 2014 at 7:55 PM, Jean-Marc Spaggiari <
> > >> [email protected]> wrote:
> > >>
> > >> > I looked a bit closer to the numbers. Just for FilteredScanTest for
> > now.
> > >> >
> > >> > Here are the results for the few last versions. Is it possible that
> > >> > something went into 0.98.5 but not into 0.99?
> > >> >
> > >> > 0.98.30.98.40.98.50.98.60.99.00,250,2567,3467,490,63
> > >> > I will re-run for 0.98.4 and 0.98.6. Numbers for 0.99.0 are already
> > >> from my
> > >> > 2nd run.
> > >> >
> > >> > JM
> > >> >
> > >> > 2014-09-20 14:28 GMT-04:00 Jean-Marc Spaggiari <
> > [email protected]
> > >> >:
> > >> >
> > >> > > Sure. I will. I will also update the numbers as soon as I have the
> > new
> > >> > > one. Will also run some other perfs tests to compare with 0.98.
> > >> > >
> > >> > > JM
> > >> > >
> > >> > > 2014-09-20 14:02 GMT-04:00 Enis Söztutar <[email protected]>:
> > >> > >
> > >> > > Agreed. Will be good to fix it in 0.99.1.
> > >> > >>
> > >> > >> On Saturday, September 20, 2014, Stack <[email protected]> wrote:
> > >> > >>
> > >> > >> > JMS file blocker against 0.99.1?
> > >> > >> > Good work boss
> > >> > >> > Stack
> > >> > >> > On Sep 19, 2014 7:00 PM, "Jean-Marc Spaggiari" <
> > >> > [email protected]
> > >> > >> > <javascript:;>>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Sorry, here is the destination:
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> https://docs.google.com/spreadsheets/d/1yo-A-f4tjchdT9R-hkh6CkcXrbBHG_K2Y_ptF9QPT1Q/edit?usp=sharing
> > >> > >> > >
> > >> > >> > > 2014-09-19 21:55 GMT-04:00 Andrew Purtell <
> [email protected]
> > >> > >> > <javascript:;>>:
> > >> > >> > >
> > >> > >> > > > On Fri, Sep 19, 2014 at 6:44 PM, Jean-Marc Spaggiari
> > >> > >> > > > <[email protected] <javascript:;>> wrote:
> > >> > >> > > > > Results are there: http://tiny.cloudera.com/sjeyn
> > >> > >> > > >
> > >> > >> > > > I get "Access not possible"
> > >> > >> > > >
> > >> > >> > > > On Fri, Sep 19, 2014 at 6:44 PM, Jean-Marc Spaggiari
> > >> > >> > > > <[email protected] <javascript:;>> wrote:
> > >> > >> > > > > UniformRandomReadBenchmark is not working anymore:
> > >> > >> > > > >
> > >> > >> > > > > 2014-09-19 05:18:54,719 INFO  [0]
> > >> > >> hbase.HFilePerformanceEvaluation:
> > >> > >> > > > Running
> > >> > >> > > > > UniformRandomReadBenchmark for 1000000 rows.
> > >> > >> > > > > 2014-09-19 05:18:54,719 INFO  [0] hfile.CacheConfig:
> > >> > >> > > CacheConfig:disabled
> > >> > >> > > > > Exception in thread "0" java.lang.AssertionError:
> Expected
> > >> > >> 0000472128
> > >> > >> > > but
> > >> > >> > > > > got 0000472127
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.PerformanceEvaluationCommons.assertKey(PerformanceEvaluationCommons.java:50)
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.PerformanceEvaluationCommons.assertKey(PerformanceEvaluationCommons.java:45)
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.HFilePerformanceEvaluation$UniformRandomReadBenchmark.doRow(HFilePerformanceEvaluation.java:295)
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.HFilePerformanceEvaluation$RowOrientedBenchmark.run(HFilePerformanceEvaluation.java:169)
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.HFilePerformanceEvaluation.runBenchmark(HFilePerformanceEvaluation.java:121)
> > >> > >> > > > >         at
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> org.apache.hadoop.hbase.HFilePerformanceEvaluation$2.run(HFilePerformanceEvaluation.java:87)
> > >> > >> > > > >         at java.lang.Thread.run(Thread.java:744)
> > >> > >> > > > >
> > >> > >> > > > > Also, on the performance side, I got very strange
> results.
> > >> > >> > > > >
> > >> > >> > > > > Some tests are as fast as 0.98.6, some are WAY slower...
> > >> > >> > > > >
> > >> > >> > > > > Results are there: http://tiny.cloudera.com/sjeyn I will
> > >> > restart
> > >> > >> the
> > >> > >> > > > tests
> > >> > >> > > > > tomorrow and get new numbers by mid next week.
> > >> > >> > > > >
> > >> > >> > > > > Appart from that, it works. And since it's a dev release,
> > >> I'm +1
> > >> > >> with
> > >> > >> > > it.
> > >> > >> > > > >
> > >> > >> > > > > JM
> > >> > >> > > > >
> > >> > >> > > > > 2014-09-19 21:28 GMT-04:00 Andrew Purtell <
> > >> [email protected]
> > >> > >> > <javascript:;>>:
> > >> > >> > > > >
> > >> > >> > > > >> +1
> > >> > >> > > > >>
> > >> > >> > > > >> Checked file sums and signature
> > >> > >> > > > >> Unpacked tarball, looked at directory layout
> > >> > >> > > > >> Started up a minicluster in tree, did a quick ingest
> test,
> > >> it
> > >> > >> > > succeeded
> > >> > >> > > > >>
> > >> > >> > > > >>
> > >> > >> > > > >> On Tue, Sep 16, 2014 at 1:52 PM, Enis Söztutar <
> > >> > [email protected]
> > >> > >> > <javascript:;>>
> > >> > >> > > wrote:
> > >> > >> > > > >> > I am very happy to say that the second HBase 0.99.0
> > >> release
> > >> > >> > > candidate
> > >> > >> > > > >> (RC1)
> > >> > >> > > > >> > is
> > >> > >> > > > >> > available for download at
> > >> > >> > > > >> https://people.apache.org/~enis/hbase-0.99.0RC1/
> > >> > >> > > > >> >
> > >> > >> > > > >> > Maven artifacts are also available in the temporary
> > >> > repository
> > >> > >> > > > >> >
> > >> > >> > > >
> > >> > >> >
> > >> >
> > https://repository.apache.org/content/repositories/orgapachehbase-1039/
> > >> > >> > > > >> >
> > >> > >> > > > >> > Signed with my code signing key E964B5FF. Can be found
> > >> here:
> > >> > >> > > > >> > https://people.apache.org/keys/committer/enis.asc
> > >> > >> > > > >> >
> > >> > >> > > > >> > Signed tag in the repository can be found here:
> > >> > >> > > > >> >
> > >> > >> > > > >>
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=336a63b5f58bb1c4859e3c2b6a2f5c0d68a7d9e1
> > >> > >> > > > >> >
> > >> > >> > > > >> > NOTE IN GIGANTIC LETTERS THAT THIS IS A DEVELOPER
> > RELEASE.
> > >> > >> > > > >> > DO NOT USE THIS RELEASE IN PRODUCTION.
> > >> > >> > > > >> >
> > >> > >> > > > >> > HBase 0.99.0 is a "developer preview" release, and an
> > >> > >> odd-numbered
> > >> > >> > > > >> release
> > >> > >> > > > >> > as
> > >> > >> > > > >> > defined in
> > >> > >> > > > >>
> > >> https://hbase.apache.org/book/upgrading.html#hbase.versioning.
> > >> > >> > > > >> > This release IS NOT intended for production use, and
> > does
> > >> not
> > >> > >> > > contain
> > >> > >> > > > any
> > >> > >> > > > >> > backwards or forwards compatibility guarantees (even
> > >> within
> > >> > >> minor
> > >> > >> > > > >> versions
> > >> > >> > > > >> > 0.99.x). Please refrain from deploying this over
> > important
> > >> > >> data.
> > >> > >> > > > >> >
> > >> > >> > > > >> > 0.99.0 release is provided from branch-1, which will
> be
> > >> the
> > >> > >> basis
> > >> > >> > > for
> > >> > >> > > > >> > HBase-1.0
> > >> > >> > > > >> > release. A few 0.99.x releases are planned before 1.0.
> > The
> > >> > >> reason
> > >> > >> > > for
> > >> > >> > > > >> > doing a developer preview release is to get more
> testing
> > >> for
> > >> > >> the
> > >> > >> > > > branch-1
> > >> > >> > > > >> > code
> > >> > >> > > > >> > that will be released soon as HBase-1.0.0. Thus, all
> > >> > >> contribution
> > >> > >> > in
> > >> > >> > > > >> terms
> > >> > >> > > > >> > of
> > >> > >> > > > >> > testing, benchmarking, checking API / source /wire
> > >> > >> compatibility,
> > >> > >> > > > >> checking
> > >> > >> > > > >> > out
> > >> > >> > > > >> > documentation and further code contribution is highly
> > >> > >> appreciated.
> > >> > >> > > 1.0
> > >> > >> > > > >> will
> > >> > >> > > > >> > be
> > >> > >> > > > >> > the first series in the 1.x line of releases which are
> > >> > >> expected to
> > >> > >> > > > keep
> > >> > >> > > > >> > compatibility with previous 1.x releases. Thus it is
> > very
> > >> > >> > important
> > >> > >> > > to
> > >> > >> > > > >> > check
> > >> > >> > > > >> > the client side and server side APIs for compatibility
> > and
> > >> > >> > > > >> maintainability
> > >> > >> > > > >> > concerns for future releases.
> > >> > >> > > > >> >
> > >> > >> > > > >> > 0.99.0 contains slightly more than 1K issues resolved
> > with
> > >> > many
> > >> > >> > > > >> > improvements
> > >> > >> > > > >> > and bug fixes. The theme of (eventual) 1.0 release is
> to
> > >> > >> become a
> > >> > >> > > > stable
> > >> > >> > > > >> > base
> > >> > >> > > > >> > for future 1.x series of releases. 1.0 release will
> aim
> > to
> > >> > >> achieve
> > >> > >> > > at
> > >> > >> > > > >> least
> > >> > >> > > > >> > the same level of stability of 0.98 releases without
> > >> > >> introducing
> > >> > >> > too
> > >> > >> > > > many
> > >> > >> > > > >> > new
> > >> > >> > > > >> > features.
> > >> > >> > > > >> >
> > >> > >> > > > >> > Some work has been under way to clearly mark and
> > >> > differentiate
> > >> > >> > > client
> > >> > >> > > > >> > facing
> > >> > >> > > > >> > APIs, and redefine some of the client interfaces for
> > >> > improving
> > >> > >> > > > semantics,
> > >> > >> > > > >> > easy
> > >> > >> > > > >> > of use and maintainability. 0.99.0 contains some of
> the
> > >> > >> completed
> > >> > >> > > > items,
> > >> > >> > > > >> > and
> > >> > >> > > > >> > the rest can be found HBASE-10602. Marking/remarking
> of
> > >> > >> interfaces
> > >> > >> > > > with
> > >> > >> > > > >> > InterfaceAudience has also been going on, which will
> > >> identify
> > >> > >> > areas
> > >> > >> > > > for
> > >> > >> > > > >> > compatibility (with clients, coprocessors and
> dependent
> > >> > >> projects
> > >> > >> > > like
> > >> > >> > > > >> > Phoenix)
> > >> > >> > > > >> > for future releases.
> > >> > >> > > > >> >
> > >> > >> > > > >> > Starting with 0.99.0, the HBase master server and
> backup
> > >> > master
> > >> > >> > > > servers
> > >> > >> > > > >> > will
> > >> > >> > > > >> > also act as a region server. RPC port and info port
> for
> > >> web
> > >> > UI
> > >> > >> is
> > >> > >> > > > shared
> > >> > >> > > > >> > for
> > >> > >> > > > >> > the master and region server roles. Active master and
> > >> backup
> > >> > >> > masters
> > >> > >> > > > by
> > >> > >> > > > >> > default
> > >> > >> > > > >> > will NOT be hosting any regions unless configured
> > >> otherwise.
> > >> > >> See
> > >> > >> > > > >> > HBASE-10569
> > >> > >> > > > >> > and HBASE-11604 for more details.
> > >> > >> > > > >> >
> > >> > >> > > > >> > This release contains Phase 1 items for experimental
> > "Read
> > >> > >> > > > availability
> > >> > >> > > > >> > using
> > >> > >> > > > >> > timeline consistent region replicas" feature. A region
> > >> can be
> > >> > >> > hosted
> > >> > >> > > > in
> > >> > >> > > > >> > multiple region servers in read-only mode. One of the
> > >> > replicas
> > >> > >> for
> > >> > >> > > the
> > >> > >> > > > >> > region
> > >> > >> > > > >> > will be primary, accepting writes, and other replicas
> > >> will be
> > >> > >> > > sharing
> > >> > >> > > > the
> > >> > >> > > > >> > same
> > >> > >> > > > >> > data files. Read requests can be done against any
> > replica
> > >> for
> > >> > >> the
> > >> > >> > > > region
> > >> > >> > > > >> > with
> > >> > >> > > > >> > backup RPCs for high availability with timeline
> > >> consistency
> > >> > >> > > > guarantees.
> > >> > >> > > > >> > More
> > >> > >> > > > >> > information can be found at HBASE-10070.
> > >> > >> > > > >> >
> > >> > >> > > > >> > Other notable improvements in this release are
> > >> > >> > > > >> >  - Automatic tuning of global memstore and block cache
> > >> sizes
> > >> > >> > > > >> >  - Various security, tags and visibility labels
> > >> improvements
> > >> > >> > > > >> >  - Bucket cache improvements (usability and compressed
> > >> data
> > >> > >> > blocks)
> > >> > >> > > > >> >  - A new pluggable replication endpoint to plug in to
> > >> HBase's
> > >> > >> > > > >> inter-cluster
> > >> > >> > > > >> >    replication to replicate to a custom data store
> > >> > >> > > > >> >  - A Dockerfile to easily build and run HBase from
> > source
> > >> > >> > > > >> >  - Internal refactoring for abstracting away zookeeper
> > >> usage
> > >> > >> > > > >> >  - Truncate table command
> > >> > >> > > > >> >  - Region assignment to use hbase:meta table instead
> of
> > >> > >> zookeeper
> > >> > >> > > for
> > >> > >> > > > >> faster
> > >> > >> > > > >> >    region assignment (disabled by default)
> > >> > >> > > > >> >  - Better support for Cell interface internally in
> read
> > >> and
> > >> > >> write
> > >> > >> > > > paths
> > >> > >> > > > >> for
> > >> > >> > > > >> >    better performance and flexibility
> > >> > >> > > > >> >  - Combining internal mvcc and seqId infrastructure
> > >> > >> > > > >> >  - Extensive documentation improvements
> > >> > >> > > > >> >  - Numerous improvements in other areas and bug fixes.
> > >> > >> > > > >> >
> > >> > >> > > > >> > The release has these changes in default behaviour:
> > >> > >> > > > >> >  - hfile version 3 has been enabled by default
> > >> > >> > > > >> >  - Distributed log replay has been enabled by default
> > >> > >> > > > >> >  - Slab cache has been removed (use bucket cache
> > instead)
> > >> > >> > > > >> >  - Some client facing APIs (HTableInterface, etc) has
> > been
> > >> > >> > > deprecated
> > >> > >> > > > and
> > >> > >> > > > >> >    replaced.
> > >> > >> > > > >> >
> > >> > >> > > > >> > We have also adopted an orca as the mascot for Apache
> > >> HBase.
> > >> > >> See
> > >> > >> > it
> > >> > >> > > at
> > >> > >> > > > >> > https://hbase.apache.org/
> > >> > >> > > > >> >
> > >> > >> > > > >> > Thanks for everybody who have contributed to this
> > release.
> > >> > Full
> > >> > >> > list
> > >> > >> > > > of
> > >> > >> > > > >> the
> > >> > >> > > > >> > issues
> > >> > >> > > > >> > can be found here:
> > >> > >> > > > >> >
> > >> > >> > > > >>
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12325675
> > >> > >> > > > >> >
> > >> > >> > > > >> >
> > >> > >> > > > >> > Compatibility
> > >> > >> > > > >> > -------------
> > >> > >> > > > >> > 0.99.x series of releases are preview of upcoming 1.0
> > >> > release.
> > >> > >> > > > HBase-1.0
> > >> > >> > > > >> is
> > >> > >> > > > >> > planned to be wire compatible with 0.98.x and 0.96.x
> > >> > releases.
> > >> > >> > > Clients
> > >> > >> > > > >> and
> > >> > >> > > > >> > servers running in different versions as long as new
> > >> features
> > >> > >> are
> > >> > >> > > not
> > >> > >> > > > >> used
> > >> > >> > > > >> > should be possible.
> > >> > >> > > > >> > A rolling upgrade from 0.98.x clusters to 0.99.0 is
> > >> supported
> > >> > >> as
> > >> > >> > > well.
> > >> > >> > > > >> > 0.99.0
> > >> > >> > > > >> > introduces a new file format (hfile v3) that is
> enabled
> > by
> > >> > >> default
> > >> > >> > > > that
> > >> > >> > > > >> > 0.96.x code cannot read. Rolling upgrade from 0.96
> > >> directly
> > >> > to
> > >> > >> > 0.99
> > >> > >> > > is
> > >> > >> > > > >> not
> > >> > >> > > > >> > tested.
> > >> > >> > > > >> >
> > >> > >> > > > >> > Direct migration from 0.94.x is also supported but it
> IS
> > >> NOT
> > >> > >> > tested.
> > >> > >> > > > >> Before
> > >> > >> > > > >> > HBase-1.0 we expect to complete the test.
> > >> > >> > > > >> >
> > >> > >> > > > >> > Binary compatibility at the Java API layer with
> earlier
> > >> > >> versions
> > >> > >> > > > (0.98.x,
> > >> > >> > > > >> > 0.96.x and 0.94.x) is not supported. You may have to
> > >> > recompile
> > >> > >> > your
> > >> > >> > > > >> client
> > >> > >> > > > >> > code and any server side code (coprocessors, filters
> > etc)
> > >> > >> > referring
> > >> > >> > > to
> > >> > >> > > > >> > hbase jars.
> > >> > >> > > > >> >
> > >> > >> > > > >> > 0.99.0 release introduces new APIs, and deprecates
> some
> > of
> > >> > >> > > > commonly-used
> > >> > >> > > > >> > client side APIs (HTableInterface, etc). However,
> client
> > >> side
> > >> > >> code
> > >> > >> > > is
> > >> > >> > > > >> > expected
> > >> > >> > > > >> > to be source compatible with earlier versions. However
> > we
> > >> > >> advise
> > >> > >> > to
> > >> > >> > > > >> update
> > >> > >> > > > >> > your application to use the new style of APIs, since
> > >> > deprecated
> > >> > >> > APIs
> > >> > >> > > > >> might
> > >> > >> > > > >> > be removed in future releases (2.x).
> > >> > >> > > > >> >
> > >> > >> > > > >> >
> > >> > >> > > > >> > Supported Hadoop versions
> > >> > >> > > > >> > -------------------------
> > >> > >> > > > >> > 0.99.0 release drops support for Hadoop-1.x releases.
> > Only
> > >> > >> > > Hadoop-2.x
> > >> > >> > > > >> > releases are supported. Hadoop-2.4.x and Hadoop-2.5.x
> > >> > releases
> > >> > >> are
> > >> > >> > > the
> > >> > >> > > > >> > most tested hadoop releases and we recommend running
> > with
> > >> > those
> > >> > >> > > > versions.
> > >> > >> > > > >> > Earlier Hadoop-2 based releases (hadoop-2.2.x and
> 2.3.x)
> > >> are
> > >> > >> not
> > >> > >> > > > tested
> > >> > >> > > > >> to
> > >> > >> > > > >> > the full
> > >> > >> > > > >> > extend. More information can be found here:
> > >> > >> > > > >> >
> https://hbase.apache.org/book/configuration.html#hadoop
> > >> > >> > > > >> >
> > >> > >> > > > >> >
> > >> > >> > > > >> > Supported Java versions
> > >> > >> > > > >> > -------------------------
> > >> > >> > > > >> > 0.99.0 release drops support for JDK6. Only JDK7 is
> > >> > supported.
> > >> > >> > JDK8
> > >> > >> > > > >> support
> > >> > >> > > > >> > is experimental. More information can be found here:
> > >> > >> > > > >> > https://hbase.apache.org/book/configuration.html#java
> > >> > >> > > > >> >
> > >> > >> > > > >> >
> > >> > >> > > > >> > Voting
> > >> > >> > > > >> > ------
> > >> > >> > > > >> > Please try to test and vote on this release by
> > 09/19/2014
> > >> > >> 11:59PM
> > >> > >> > > PDT.
> > >> > >> > > > >> > This is a developer (beta) release, so we have a
> shorter
> > >> vote
> > >> > >> > time.
> > >> > >> > > > >> >
> > >> > >> > > > >> > [] +1 Release the artifacts as 0.99.0
> > >> > >> > > > >> > [] -1 DO NOT release the artifacts as 0.99.0,
> because...
> > >> > >> > > > >> >
> > >> > >> > > > >> > Thanks for helping to get a stable release out!
> > >> > >> > > > >> >
> > >> > >> > > > >> > Enis
> > >> > >> > > > >>
> > >> > >> > > > >>
> > >> > >> > > > >>
> > >> > >> > > > >> --
> > >> > >> > > > >> Best regards,
> > >> > >> > > > >>
> > >> > >> > > > >>    - Andy
> > >> > >> > > > >>
> > >> > >> > > > >> Problems worthy of attack prove their worth by hitting
> > >> back. -
> > >> > >> Piet
> > >> > >> > > > >> Hein (via Tom White)
> > >> > >> > > > >>
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > --
> > >> > >> > > > Best regards,
> > >> > >> > > >
> > >> > >> > > >    - Andy
> > >> > >> > > >
> > >> > >> > > > Problems worthy of attack prove their worth by hitting
> back.
> > -
> > >> > Piet
> > >> > >> > > > Hein (via Tom White)
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Reply via email to