Re: Relying more on Mouse

[Ovid]

>
> From: Michael G Schwern 
>
>On 2011.11.21 4:07 AM, David Cantrell wrote:
>> But then how often does one need to 'use Test::More'?  Not enough to
>> bother optimising it, I'd say.
>
>In every single .t file that gets run by just about everybody.
>
>By being THE testing framework, it places an upper bound on how fast anyone's
>tests can be.  10 .t files per second, no faster.  That sucks.
>
>
>> To take a real-world example, it occurs 182 times in our test suite at
>> work, a test suite that takes almost 2 hours to run in full.  Those
>> extra 182 * 0.07 == 13 seconds are of absolutely no importance at all.
>
>I run tests a lot while developing.  I can see they're slower without even
>benchmarking it.  You wouldn't think you'd notice 70 ms, but you do.
>
>I don't want to give anyone an excuse to not run tests often or not upgrade.


I have to agree with this. Test performance is an issue that people constantly 
hit. And aside from Test::Class/Test::Aggregate style solutions (I see to 
recall someone put out an alternative to Test::Aggregate?), individual tests in 
separate processes is the norm. 

For larger, "enterprisey" type systems, my experience shows that they generally 
have enough other issues that a tiny hit on Test::More won't matter, but when 
you add up all the tiny hits, it's death by a thousand cuts. Dave, you weren't 
on the PIPs team when I optimized their test suite, but it was those "thousand 
cuts" that I stripped away one by one to get an hour long test suite running in 
less than 15 minutes.

So yeah, this is a very important issue. If performance isn't dealt with, it's 
going to be a constant sore spot.

And Schwern: thanks for all of your good work in this area!

Cheers,
Ovid
--
Live and work overseas - http://overseas-exile.blogspot.com/
Buy the book   - http://www.oreilly.com/catalog/perlhks/
Tech blog  - http://blogs.perl.org/users/ovid/
Twitter- http://twitter.com/OvidPerl/

Threads working, Test::Builder1.5 is feature complete

[Michael G Schwern]

Threads are now working in Test::Builder1.5.  That's the last missing feature,
Test::Builder1.5 can now do everything 0.98 can!  Huzzah!

If anyone want to see how much working with threads suck, just have a look at
this hack.
https://github.com/schwern/test-more/blob/Test-Builder1.5/lib/TB2/ThreadSafeFilehandleAccessor.pm

Anyhow, there's lots of stuff to do before stable.  Some of it is in
Test::Builder2, but a lot of it is fixing broken test modules that violated
encapsulation. :-/
https://github.com/schwern/test-more/issues?milestone=6

There's a bug in use_ok() that effects 0.98_01 and 1.5.  So I'm going to hold
off on a new alpha for a day or two and it's either fixed or I'll roll it back.
https://rt.cpan.org/Ticket/Display.html?id=67538#txn-1002509


-- 
124. Two drink limit does not mean first and last.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: Relying more on Mouse

[Buddy Burden]

On Mon, Nov 21, 2011 at 5:35 AM, yary  wrote:
> I'd think Michael has the interests of CPAN smoke testers in mind with
> these performance benchmarks. You're right in that for the typical
> developer, it's not significant.

Just to offer a contrasting viewpoint: if you're using TDD, you're
running tests *constantly*.  My typical work cycle is to run a single
test file (whichever one I'm working on) over and over--several times
a minute--until I get to a certain point, then run a larger subset of
tests to verify I haven't broken anything (perhaps a couple times an
hour), then the full test suite before I check anything in.  So the
test speed is pretty important.  The slower the tests run, the more it
encourages people to delay running the tests, which in turn means more
changes between test runs and longer debugging cycles.


            -- Buddy

Re: Threads working, Test::Builder1.5 is feature complete

[Ovid]

- Original Message -

> From: Michael G Schwern 


> There's a bug in use_ok() that effects 0.98_01 and 1.5.  So I'm going to 
> hold
> off on a new alpha for a day or two and it's either fixed or I'll roll 
> it back.
> https://rt.cpan.org/Ticket/Display.html?id=67538#txn-1002509


Ah, just saw this. As I've already said privately, but maybe we can see how 
others feel, this is a PERFECT time to discourage use_ok and require_ok and 
even deprecate them (though I doubt we can remove them completely).

They've been broken for a long time, people have to work around their 
limitations, and they don't add value.

http://use.perl.org/~Ovid/journal/39859

I'm hard-pressed to think of a better time to at least slip a note in the docs 
that their use is discouraged.

Cheers,
Ovid

--
Live and work overseas - http://overseas-exile.blogspot.com/
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog - http://blogs.perl.org/users/ovid/
Twitter - http://twitter.com/OvidPerl/

Re: QA hackathon 2012

[Philippe Bruhat (BooK)]

On Tue, Aug 23, 2011 at 04:48:58PM +0100, Miss Barbie wrote:
> On Tue, Aug 23, 2011 at 01:07:36AM +0200, Lars Dɪᴇᴄᴋᴏᴡ
> 迪拉斯 wrote:
> > To the usual suspects on this list, especially the ones who could
> > not attend the last times: when do you have time around April?
> > (Traditionally, the QA hackathon is from Saturday to Monday and
> > avoids Easter.)
>
> I hope to attend next year, but depends on any touring commitments I
> may have. The band seem to like April and October for their gigs :)
>
> Whoever ends up organising the 2012 and 2013, can you let me know the
> appropriate IP addresses, so I can set up the DNS for the *.qa-
> hackathon.org entries :)
>

The site is up (with an ugly logo left as an incentive for volunteers to
improve the site looks), and will reply to http://2012.qa-hackathon.org/
as soon as the DNS are updated (same IP as for 2011).

daxim and mdk, you have commit bits. It's just another Act site.

-- 
 Philippe Bruhat (BooK)

 In the contest between simplicity and silence, silence hasn't got a prayer.
(Moral from Groo The Wanderer #15 (Epic))

Re: Relying more on Mouse

[Eric Wilhelm]

# from Michael G Schwern
# on Monday 21 November 2011 17:56:

>In every single .t file that gets run by just about everybody.
>
>By being THE testing framework, it places an upper bound on how fast
> anyone's tests can be.  10 .t files per second, no faster.  That
> sucks.

I agree.  But, with XS mouse, you're only cutting the startup time to 
0.07s from 0.09s, correct?  And really, 14.3 .t/s also sucks vs the 
50 .t/s with v0.98, no?

Is there a way to remove some of the work Mouse is doing at startup?  
What is it doing?

I did once suggest a pre-loading / forking harness (along the lines of 
Test::Aggregate), though it would probably be much better to make the 
test code start faster rather than need to work around it.

--Eric
-- 
You can't whack a chisel without a big wooden mallet.
---
http://scratchcomputing.com
---

Re: Relying more on Mouse

[David Golden]

On Tue, Nov 22, 2011 at 2:02 PM, Eric Wilhelm  wrote:
> Is there a way to remove some of the work Mouse is doing at startup?
> What is it doing?

How much of Mouse is needed?  Could Moo be used?  (I ask without
having read the details of the OO breakdown of TB2)

-- David

Re: QA hackathon 2012

[Barbie]

On Tue, Nov 22, 2011 at 12:47:19PM +0100, Philippe Bruhat (BooK) wrote:
> 
> The site is up (with an ugly logo left as an incentive for volunteers to
> improve the site looks), and will reply to http://2012.qa-hackathon.org/
> as soon as the DNS are updated (same IP as for 2011).

DNS should hopefully be fully propagated by tomorrow.

Cheers,
Barbie.
-- 
Birmingham Perl Mongers 
Memoirs Of A Roadie 
CPAN Testers Blog 
YAPC Conference Surveys 
Ark Appreciation Pages 

Re: Relying more on Mouse

[Michael G Schwern]

On 2011.11.22 11:22 AM, David Golden wrote:
> On Tue, Nov 22, 2011 at 2:02 PM, Eric Wilhelm  wrote:
>> Is there a way to remove some of the work Mouse is doing at startup?
>> What is it doing?
> 
> How much of Mouse is needed?  Could Moo be used?  (I ask without
> having read the details of the OO breakdown of TB2)

No.  Moo has runtime dependencies, is harder to bundle (multiple files), is
not upward compatible with Moose (their type system is incompatible), lacks
Moose types, a class can't use more than one role at once, and has no meta
system.  And it's slower at runtime.

I'm also not convinced of Moo's stability.  Mouse has strictly adhered to
being a subset of Moose.  Moo hasn't.  The Mouse folks have said they'd test
vs Test::Builder2.  Finally, Moo has only existed for a year.

About the only advantage is Moo loads about 10ms faster than Mouse/XS.

The major Mo[ou]se features that Test::Builder2 uses heavily is roles and
attributes.  It also uses types, coercion, meta classes and method modifiers,
but those could be removed if necessary.  I've tried to keep it simple to
avoid touching any actual or potential Mouse bugs.


-- 
There will be snacks.

Re: Relying more on Mouse

[David Golden]

On Tue, Nov 22, 2011 at 10:49 PM, Michael G Schwern  wrote:
> The major Mo[ou]se features that Test::Builder2 uses heavily is roles and
> attributes.  It also uses types, coercion, meta classes and method modifiers,
> but those could be removed if necessary.  I've tried to keep it simple to
> avoid touching any actual or potential Mouse bugs.

Attributes -- leaving aside types and coercion -- are you just talking
about the sugar?

And I wonder if Role::Tiny could be extended to do multiple roles.

Or could you do a poor-man's version of roles using mix-ins?

I understand the goal of decomposition and why Mo[ou]se rocks, but for
a testing framework, maybe simpler is better.

For example, Moose will detect method clashes with roles, right?  (I
assume Mouse does that.)  Do you really need that?  Or could you
verify the design "by hand" rather than relying on a MOP to do the
work for you?

How much of it is fundamental to what you're trying to do and how much
of it is just making life easier for you in implementing it?  Granted,
well tested Mouse might be less buggy than hand-rolled replacement,
but a special-purpose class framework could possibly be simpler (thus
faster and less buggy).

[N.B. Not arguing not to use it -- trying to provoke discussion of
preexisting assumptions.]

-- David

Re: Relying more on Mouse

[Michael G Schwern]

On 2011.11.22 11:02 AM, Eric Wilhelm wrote:
>> By being THE testing framework, it places an upper bound on how fast
>> anyone's tests can be.  10 .t files per second, no faster.  That
>> sucks.
> 
> I agree.  But, with XS mouse, you're only cutting the startup time to 
> 0.07s from 0.09s, correct?  And really, 14.3 .t/s also sucks vs the 
> 50 .t/s with v0.98, no?

That's before any further optimization is done.  XS Mouse buys us 20ms before
we even start.


> Is there a way to remove some of the work Mouse is doing at startup?  
> What is it doing?

I don't know about Mouse, but I know Test::Builder2 is eating 13ms building up
all the different types of Results at compile time in TB2::Result.  Results
are over-engineered.  I have a not-too-old branch to simplify them.

Some of the time is spent applying roles and generating accessors and there
isn't much to be done about that in Test::Builder2.  Performance enhancements
in Mouse itself would be appreciated by all.

I've added a branch called "gonzales" which uses Mouse directly.  You can
experiment with that and NYTProf to examine compile time performance issues
yourself.
https://github.com/schwern/test-more/commits/gonzales


> I did once suggest a pre-loading / forking harness (along the lines of 
> Test::Aggregate), though it would probably be much better to make the 
> test code start faster rather than need to work around it.

That's an interesting idea.

FWIW Test::Aggregate should become much easier with Test::Builder2.  Instead
of the shenanigans it goes through to merge tests together, it can just
intercept set_plan events and turn them into subtests or just scrub them out
entirely.


-- 
184. When operating a military vehicle I may *not* attempt something
 "I saw in a cartoon".
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/