[issue22197] Allow better verbosity / output control in test cases

[Michael Foord]

Michael Foord added the comment:

Using the runner as a "context" passed to test cases (and accessible from 
tests) for this kind of configuration seems like a good approach to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[R. David Murray]

R. David Murray added the comment:

I like that approach as well, it is the kind of API I've been finding myself 
writing a fair bit lately (a context passed to class instances that otherwise 
don't store global state themselves).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Robert Collins]

Robert Collins added the comment:

There's a few interacting things here. If I can suggest some design thoughts.

buffering within a test is I think really something we should offer a test 
servicing API for. There are many thirdparty ones (e.g. I have one in fixtures) 
- but it should be a dedicated API, even if the implementation happens to be 
shared with the buffer runner feature.

I think it is reasonable in some cases to change TestCase behaviour based on 
how much debugging is desired - even in CI environments, some outputs are much 
more expensive than others to generate (e.g. taking a full dump of a database). 
To enable that we do need a CLI - runtime environment communication mechanism.

There are many related APIs which we could look at for doing this.
* TestCase's load API (__init__) - we'd get at this via TestLoader
* TestCase's execution API (run, debug) - we'd get at this through TestRunner 
(for more parameters) or perhaps TestResult (by adding some attribute/query 
method).
* TestCase's test servicing API (assert*) - we could add a query method or a 
well known attribute to abstract out where the info is actually coming from
* TestCase's user supplied API (setUp, test methods)
* TestResult - constructed by the runner, it is in some ways the natural means 
for passing stuff like this around - its how result does its buffering today: 
but IMO the implementation is in the wrong spot there [it precludes 
parallelisation vs a test service API that uses a mutex only when needed], so 
I'd like to not repeat that here.

TestCases support being run multiple times, so to me that rules out using the 
load API.

I very much want to take TestResult in the direction of being forward-only: 
feed-backwards APIs on it are much more complex to reason about in the case of 
Y joins in the result chain. So I think its time we introduced a runner - test 
API.

Something like

def run(self, result=None, runner=None):
   ...
   dunder_set(self, '__unittest__runner__', runner)
   try:
   ...
   finally:
   dunder_set(self, '__unittest__runner__', None)
   
def debug(self, runner=None):
   ... same ...

with a helper to pull that out without folk's own code triggering name mangling.

And runner would then be the home for things like aborting a test run, 
providing configuration like verbosity, arbitrary end user options and so on.
Something like:
runner.config = {}
runner.config['verbosity'] = ...
runner.config['user'] = {... marshalled from CLI ... }

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Demian Brecht]

Changes by Demian Brecht demianbre...@gmail.com:


--
nosy:  -demian.brecht

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +rbcollins

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is the verbose attribute of the test.support module.

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Ezio Melotti]

Ezio Melotti added the comment:

That only works for the CPython test suite (and it's not a public API).

FWIW I'm +1 on the idea, but I would have to see how it will get implemented in 
a patch.

--
stage:  - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Usages of test.support.verbose should be replaced by self.verbosity.

As for output buffering, may be replace sys.stdout by file-like object which 
flushes its buffered content to original stdout on failure and discard it on 
success. Or add the self.log file-like object with such behavior and redirect 
all verbose output to it.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Ezio Melotti]

Ezio Melotti added the comment:

 As for output buffering, may be replace sys.stdout by file-like object
 which flushes its buffered content to original stdout on failure and
 discard it on success.

This is what the --buffer option is already supposed to do (I only found out 
about it thanks to this issue, the name is not very indicative of what it 
does...).  IIUC what Antoine is suggesting is having a more fine-grained 
control of the buffering, and the ability to set it from individual test cases 
rather than using a global command line flag or unittest.main(buffer=True) 
(which is only used while executing the test file directly).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Antoine Pitrou]

Antoine Pitrou added the comment:

Indeed, I'm sorry for suggesting two features in one issue :-)

Feature #1 is self.verbosity (as a read-only variable) on test cases. Sounds 
like a no-brainer, IMHO :-)

Feature #2 is selective enabling of the buffering feature in test cases. This 
rather misnamed features only prints out stdout when the test fails, which is 
useful when you want permanent debug statements that only pollute stdout when 
there is a test failure.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Michael Foord]

Michael Foord added the comment:

This seems like a reasonable improvement, I'd be in favour. I'd be *slightly* 
concerned that a test can override an explicit verbosity setting of the user, 
but I guess that annoyance is up to the test writer (they should check for an 
explicit setting first).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Antoine Pitrou]

Antoine Pitrou added the comment:

Oh, I'm only proposing read-only access to the verbosity information. And 
forcing output buffering would only last for the current test method, of course.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Michael Foord]

Michael Foord added the comment:

Yep. I have no qualms about allowing test cases to switch on buffering.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Berker Peksag]

Changes by Berker Peksag berker.pek...@gmail.com:


--
nosy: +berker.peksag

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Demian Brecht]

Changes by Demian Brecht demianbre...@gmail.com:


--
nosy: +demian.brecht

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Antoine Pitrou]

New submission from Antoine Pitrou:

Currently, test cases have no control over output and verbosity. I would 
propose two possible enhancements:
- give the TestCase read access to the verbosity value (as e.g. 
`self.verbosity`), to allow for conditional output in tests
- allow test methods to force output buffering (rather than only let the user 
specify it on the command line), because you would like some stuff to only be 
printed out on failure; a decorator would IMO be fine

--
components: Library (Lib)
messages: 225304
nosy: ezio.melotti, michael.foord, pitrou
priority: normal
severity: normal
status: open
title: Allow better verbosity / output control in test cases
type: enhancement
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue22197] Allow better verbosity / output control in test cases

[Zachary Ware]

Changes by Zachary Ware zachary.w...@gmail.com:


--
nosy: +zach.ware

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue22197
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com