Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Am 03.11.2010 03:35, schrieb Antoine Pitrou: On Tue, 2 Nov 2010 19:57:48 -0700 Brett Cannon br...@python.org wrote: How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -1. unittest.TestCase is far simpler and more obvious that any javaesque qualified name. urllib.request and friends are already annoying enough. Agreed. There are about 30 names importable from unittest, that is quite manageable in a single namespace. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 02:57, Brett Cannon wrote: On Tue, Nov 2, 2010 at 19:50, Michael Foordfuzzy...@voidspace.org.uk wrote: On 02/11/2010 02:35, Brett Cannon wrote: On Wed, Oct 27, 2010 at 03:42, Antoine Pitrousolip...@pitrou.netwrote: On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolskyalexander.belopol...@gmail.comwrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). But not the public API as documented, e.g., it's documented as unittest.TestCase, not unittest.case.TestCase as the file structure suggests. Right. I don't think that whether or not the unittest package structure is a good structure or not is determined by where we make users import the names from. Like others I see little value in reccommending people use the longer form for imports. All the best, Michael Foord How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -Brett He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward. [snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly. It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score. As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post. All the best, Michael Foord I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 11/03/2010 01:47 AM, Ben Finney wrote: If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. I understand this reasoning, but I'd like to offer counter-examples. For instance, would you say that glob.glob0 and glob.glob1 are public API? They're undocumented, they're not in __all__, but they don't have a leading underscore either, and source comments call them helper functions. I'm sure there is a lot of other examples like that, both in the standard library and in python packages out there. Other than the existing practice, there is the matter of esthetics. Accepting underscore-less identifiers as automatically public leads to a proliferation of identifiers with leading underscores, which many people (myself included) plainly don't like. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 14:05, Nick Coghlan wrote: On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... Michael Cheers, Nick. -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mercredi 03 novembre 2010 à 14:16 +, Michael Foord a écrit : On 03/11/2010 14:05, Nick Coghlan wrote: On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... unittest.TestCase is not really pickleable. There were test_multiprocessing issues because of that (see recent SVN checkins). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 14:17, Antoine Pitrou wrote: Le mercredi 03 novembre 2010 à 14:16 +, Michael Foord a écrit : On 03/11/2010 14:05, Nick Coghlan wrote: On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Can Python 2.7 pickles containing unittest classes be unpickled using 2.6 or earlier? Even if nobody uses the new names for imports, I believe they implicitly end up included in any pickles involving affected classes (I seem to recall we've been bitten by that before when moving things around). Yes, since unittest.TestCase is still available (as are all the names). I believe so anyway... unittest.TestCase is not really pickleable. There were test_multiprocessing issues because of that (see recent SVN checkins). Interesting. We made some fixes before 2.7 to ensure they were copyable, but we fixed this in the copy module. TestCase instances now store some method objects in a dictionary which may make them unpickleable, so that could be a new problem. I'll test with 2.6 and 2.7 to see. An easy fix would be to store the method names rather than the method objects themself (if this is indeed the cause of the problem). This is what unittest2 does so that it works with earlier versions of Python that don't have the fix we put in copy. Michael Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mercredi 03 novembre 2010 à 14:26 +, Michael Foord a écrit : Interesting. We made some fixes before 2.7 to ensure they were copyable, but we fixed this in the copy module. TestCase instances now store some method objects in a dictionary which may make them unpickleable, so that could be a new problem. I'll test with 2.6 and 2.7 to see. I don't think it is a problem in unittest, unless pickling TestCase objects is really useful. I have fixed the problem in test_multiprocessing instead. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 14:26, Michael Foord wrote:
On 03/11/2010 14:17, Antoine Pitrou wrote:
Le mercredi 03 novembre 2010 à 14:16 +, Michael Foord a écrit :
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
Sounds like a decision to split a module into a package is a big
commitment. Each of the individual file names becomes a permanent
part of the API. Even future additional splits are precluded
because it might break someones dotted import (i.e. not a single
function can be moved between those files -- once in
unittest.utils, alway in unittest.utils).
Can Python 2.7 pickles containing unittest classes be unpickled using
2.6 or earlier? Even if nobody uses the new names for imports, I
believe they implicitly end up included in any pickles involving
affected classes (I seem to recall we've been bitten by that before
when moving things around).
Yes, since unittest.TestCase is still available (as are all the names).
I believe so anyway...
unittest.TestCase is not really pickleable. There were
test_multiprocessing issues because of that (see recent SVN checkins).
Interesting. We made some fixes before 2.7 to ensure they were
copyable, but we fixed this in the copy module. TestCase instances now
store some method objects in a dictionary which may make them
unpickleable, so that could be a new problem. I'll test with 2.6 and
2.7 to see.
An easy fix would be to store the method names rather than the method
objects themself (if this is indeed the cause of the problem). This is
what unittest2 does so that it works with earlier versions of Python
that don't have the fix we put in copy.
Yep, looks like 2.7 introduced a bug making it impossible to pickle
TestCase instances. I think it will be easy to fix, I'll create a
specific issue:
$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type help, copyright, credits or license for more information.
from unittest import TestCase
from pickle import dumps
t = TestCase('assert_')
dumps(t)
ccopy_reg\n_reconstructor\np0\n(cunittest\nTestCase\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'_testMethodDoc'\np6\nS'Fail
the test unless the expression is
true.'\np7\nsS'_testMethodName'\np8\nS'assert_'\np9\nsb.
bigmac:beta.python.org michael$ python2.7
ActivePython 2.7.0.1 (ActiveState Software Inc.) based on
Python 2.7 (r27:82500, Jul 4 2010, 13:58:56)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type help, copyright, credits or license for more information.
from unittest import TestCase
from pickle import dumps
t = TestCase('assert_')
dumps(t)
Traceback (most recent call last):
File stdin, line 1, in module
File
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py,
line 1374, in dumps
Pickler(file, protocol).dump(obj)
...
File
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py,
line 306, in save
rv = reduce(self.proto)
File
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py,
line 70, in _reduce_ex
raise TypeError, can't pickle %s objects % base.__name__
TypeError: can't pickle instancemethod objects
All the best,
Michael
Michael
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.voidspace.org.uk/
READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
Sounds like a decision to split a module into a package is a big
commitment. Each of the individual file names becomes a permanent
part of the API. Even future additional splits are precluded because
it might break someones dotted import (i.e. not a single function can
be moved between those files -- once in unittest.utils, alway in
unittest.utils).
Can Python 2.7 pickles containing unittest classes be unpickled using
2.6 or earlier? Even if nobody uses the new names for imports, I
believe they implicitly end up included in any pickles involving
affected classes (I seem to recall we've been bitten by that before
when moving things around).
Yes, since unittest.TestCase is still available (as are all the names).
I believe so anyway...
Actually I think the answer is no (assuming you could pickle a
TestCase). Here's an example with TestLoader:
$ python27
Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2
Type help, copyright, credits or license for more information.
import unittest
x = unittest.TestLoader()
import pickle
pickle.dumps(x)
'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
$ python24
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type help, copyright, credits or license for more information.
import pickle
pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.')
Traceback (most recent call last):
File stdin, line 1, in ?
File /usr/lib/python2.4/pickle.py, line 1394, in loads
return Unpickler(file).load()
File /usr/lib/python2.4/pickle.py, line 872, in load
dispatch[key](self)
File /usr/lib/python2.4/pickle.py, line 1104, in load_global
klass = self.find_class(module, name)
File /usr/lib/python2.4/pickle.py, line 1138, in find_class
__import__(module)
ImportError: No module named loader
The problem is that there is no unittest.loader in 2.4, and
unittest.loader.TestLoader is the name that the 2.7 pickle creates. We
see this problem every time we try and move anything in the stdlib.
--
Eric.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 03, 2010, at 12:34 AM, Antoine Pitrou wrote: I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). +1 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 14:53, Eric Smith wrote:
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
Sounds like a decision to split a module into a package is a big
commitment. Each of the individual file names becomes a permanent
part of the API. Even future additional splits are precluded because
it might break someones dotted import (i.e. not a single function can
be moved between those files -- once in unittest.utils, alway in
unittest.utils).
Can Python 2.7 pickles containing unittest classes be unpickled using
2.6 or earlier? Even if nobody uses the new names for imports, I
believe they implicitly end up included in any pickles involving
affected classes (I seem to recall we've been bitten by that before
when moving things around).
Yes, since unittest.TestCase is still available (as are all the names).
I believe so anyway...
Actually I think the answer is no (assuming you could pickle a
TestCase). Here's an example with TestLoader:
Ah dammit, I read the question the other way round.
Michael
$ python27
Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2
Type help, copyright, credits or license for more information.
import unittest
x = unittest.TestLoader()
import pickle
pickle.dumps(x)
'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
$ python24
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type help, copyright, credits or license for more information.
import pickle
pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.')
Traceback (most recent call last):
File stdin, line 1, in ?
File /usr/lib/python2.4/pickle.py, line 1394, in loads
return Unpickler(file).load()
File /usr/lib/python2.4/pickle.py, line 872, in load
dispatch[key](self)
File /usr/lib/python2.4/pickle.py, line 1104, in load_global
klass = self.find_class(module, name)
File /usr/lib/python2.4/pickle.py, line 1138, in find_class
__import__(module)
ImportError: No module named loader
The problem is that there is no unittest.loader in 2.4, and
unittest.loader.TestLoader is the name that the 2.7 pickle creates. We
see this problem every time we try and move anything in the stdlib.
--
http://www.voidspace.org.uk/
READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 11/3/10 10:53 AM, Eric Smith wrote: The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib. And BTW: for me, this is the strongest reason not to break up modules into packages or otherwise reorganize the stdlib. -- Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 6:58 PM, Guido van Rossum gu...@python.org wrote: .. To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? I would not call it offensive, but what I find annoying is import unittest unittest.TestCase.__module__ 'unittest.case' This may not be a problem for smart tools, but for me and a simple editor what used to be: Let's find code for unittest.TestCase. 1. Open Lib/unittest.py. 2. Search for class TestCase. is now 1. Open Lib/unittest.py - No such file or directory. 2. OK, I'm in 2.7. Open Lib/unittest/__init__.py 3. Search for class TestCase - beep 4. OK, search for TestCase - from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf, .. 5. Hmm, what is .case. Ah, the relative import - open case.py 7. Search for class TestCase. 8. What exactly was I looking for? The above is only slightly exaggerated scenario that I went through several times when I started using 2.7 before I conditioned myself to grep in Lib/unittest/*.py. What is unfortunate is that file split was accompanied by an explosion of assert* methods in TestCase API which means that anyone reading 2.7 unittests is likely to encounter an unfamiliar method that has to be looked up. I think the problem that I described is just a slightly reworded problem that Raymond reported at the beginning of this thread. In other words, I am not alone in seeing this as a problem. PS: For a made from scratch API I would prefer TestCase only be available from unittest.case. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 3, 2010, at 11:25 AM, Eric Smith wrote: On 11/3/10 10:53 AM, Eric Smith wrote: The problem is that there is no unittest.loader in 2.4, and unittest.loader.TestLoader is the name that the 2.7 pickle creates. We see this problem every time we try and move anything in the stdlib. And BTW: for me, this is the strongest reason not to break up modules into packages or otherwise reorganize the stdlib. This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 3, 2010, at 1:04 PM, James Y Knight wrote: This is the strongest reason why I recommend to everyone I know that they not use pickle for storage they'd like to keep working after upgrades [not just of stdlib, but other 3rd party software or their own software]. :) +1. Twisted actually tried to preserve pickle compatibility in the bad old days, but it was impossible. Pickles should never really be saved to disk unless they contain nothing but lists, ints, strings, and dicts. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 14:53, Eric Smith wrote:
On 11/3/10 10:16 AM, Michael Foord wrote:
On 03/11/2010 14:05, Nick Coghlan wrote:
On Wed, Nov 3, 2010 at 9:32 AM, Raymond Hettinger
raymond.hettin...@gmail.com wrote:
Sounds like a decision to split a module into a package is a big
commitment. Each of the individual file names becomes a permanent
part of the API. Even future additional splits are precluded because
it might break someones dotted import (i.e. not a single function can
be moved between those files -- once in unittest.utils, alway in
unittest.utils).
Can Python 2.7 pickles containing unittest classes be unpickled using
2.6 or earlier? Even if nobody uses the new names for imports, I
believe they implicitly end up included in any pickles involving
affected classes (I seem to recall we've been bitten by that before
when moving things around).
Yes, since unittest.TestCase is still available (as are all the names).
I believe so anyway...
Actually I think the answer is no (assuming you could pickle a
TestCase). Here's an example with TestLoader:
It is actually fixable by temporarily switching the __module__ attribute
of the classes inside a __reduce__ or __reduce_ex__ method. I couldn't
see a cleaner way of doing it using the pickling protocol methods. I
asked on #python-dev but the *only* person who claimed to understand the
pickle protocol methods was Barry, and he is clearly insane.
Antoine is firmly of the opinion that making TestCase instances
unpickleable is a feature...
Although in practise this is less likely to be an issue for TestCase
directly as it is extremely rare to use them without subclassing. More
likely to be an issue for the test result or runner objects.
All the best,
Michael Foord
$ python27
Python 2.7.0+ (release27-maint:85878, Oct 28 2010, 06:40:25)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2
Type help, copyright, credits or license for more information.
import unittest
x = unittest.TestLoader()
import pickle
pickle.dumps(x)
'ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.'
$ python24
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type help, copyright, credits or license for more information.
import pickle
pickle.loads('ccopy_reg\n_reconstructor\np0\n(cunittest.loader\nTestLoader\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.')
Traceback (most recent call last):
File stdin, line 1, in ?
File /usr/lib/python2.4/pickle.py, line 1394, in loads
return Unpickler(file).load()
File /usr/lib/python2.4/pickle.py, line 872, in load
dispatch[key](self)
File /usr/lib/python2.4/pickle.py, line 1104, in load_global
klass = self.find_class(module, name)
File /usr/lib/python2.4/pickle.py, line 1138, in find_class
__import__(module)
ImportError: No module named loader
The problem is that there is no unittest.loader in 2.4, and
unittest.loader.TestLoader is the name that the 2.7 pickle creates. We
see this problem every time we try and move anything in the stdlib.
--
http://www.voidspace.org.uk/
READ CAREFULLY. By accepting and reading this email you agree,
on behalf of your employer, to release me from all obligations
and waivers arising from any and all NON-NEGOTIATED agreements,
licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
confidentiality, non-disclosure, non-compete and acceptable use
policies (”BOGUS AGREEMENTS”) that I have entered into with your
employer, its partners, licensors, agents and assigns, in
perpetuity, without prejudice to my ongoing rights and privileges.
You further represent that you have the authority to release me
from any BOGUS AGREEMENTS on behalf of your employer.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, 03 Nov 2010 19:26:53 + Michael Foord fuzzy...@voidspace.org.uk wrote: Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 03 Nov 2010 19:26:53 + Michael Foord fuzzy...@voidspace.org.uk wrote: Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine. Splitting groups of tests to run in parallel via multiple processes is a pretty good use case. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mercredi 03 novembre 2010 à 15:48 -0400, Jesse Noller a écrit : On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrou solip...@pitrou.net wrote: On Wed, 03 Nov 2010 19:26:53 + Michael Foord fuzzy...@voidspace.org.uk wrote: Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine. Splitting groups of tests to run in parallel via multiple processes is a pretty good use case. Indeed, but it implies a lot of things about TestCase instances, which could have additional non-pickleable attributes (e.g. file objects). You'd better pickle the TestCase class instead, or simply the module name as we do with regrtest -jN. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 03/11/2010 19:48, Jesse Noller wrote: On Wed, Nov 3, 2010 at 3:45 PM, Antoine Pitrousolip...@pitrou.net wrote: On Wed, 03 Nov 2010 19:26:53 + Michael Foordfuzzy...@voidspace.org.uk wrote: Antoine is firmly of the opinion that making TestCase instances unpickleable is a feature... Apparently you didn't really understand me. I'm of the opinion that making TestCase instances pickleable is useless if that pickling doesn't have well-defined semantics. And I wonder what the semantics of pickling a TestCase could be, and what the use cases are. Regards Antoine. Splitting groups of tests to run in parallel via multiple processes is a pretty good use case. That's something I've been thinking about a lot (and talking to Holger about) for the unittest plugins. I definitely won't be doing it with pickles but as Antoine says, sending test names to the subprocesses. You really want tests run in a child process to behave differently and it makes sense to set them up inside the child process. All the best, Michael Foord ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 3, 2010, at 11:26 AM, Alexander Belopolsky wrote: This may not be a problem for smart tools, but for me and a simple editor what used to be: Maybe this is the real problem? It's 2010, we should all be far enough beyond EDLIN that our editors can jump to the definition of a Python class. Even Vim can be convinced to do this (http://rope.sourceforge.net/ropevim.html). Could Python itself make this easier? Maybe ship with a command that says hey, somewhere on sys.path, there is a class with this name. Please run '$EDITOR file +line' (or the current OS's equivalent) so I can look at the source code. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Nov 3, 2010 at 4:59 PM, Glyph Lefkowitz gl...@twistedmatrix.com wrote: .. Maybe ship with a command that says hey, somewhere on sys.path, there is a class with this name. Please run '$EDITOR file +line' (or the current OS's equivalent) so I can look at the source code. Well, we already have inspect.findsource() for that. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon br...@python.org wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 15:33, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon br...@python.org wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes. -Brett Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. from unitest.case import SkipTest. Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Raymond___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 3:33 PM, Nick Coghlan wrote: On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon br...@python.org wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? I'll propose some PEP 8 wording in the bug tracker (essentially advice on when and how to use packaging), and everyone can offer their assent, dissent, and word-smithing. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 02, 2010, at 03:43 PM, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:33, Nick Coghlan ncogh...@gmail.com wrote: On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannon br...@python.org wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes. If SHOULD not MUST, then +0 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. from unitest.case import SkipTest. Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 15:47 -0700, Raymond Hettinger a écrit : What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? I don't think it's frozen. It's just that Michael and Benjamin (perhaps others too) prefer it like that, and given who does most of the maintenance and improvement work it is reasonable to respect that decision. If one day someone else becomes maintainer of unittest, then, sure, they can undo the splitting or do it differently. But right now there's no reason to change. Oh, and I much prefer a splitting without any impact on the public API. I *hate* writing urllib.request.urlopen and I really wish we hadn't done that; urllib.urlopen was so much easier to remember it isn't funny :/ Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. from unitest.case import SkipTest. I wouldn't say it was done badly, just non-standard. I was able to figure out where all the key classes were based on the file names. I personally would have no trouble doing ``from unittest.case import TestCase`` if more test case classes came along for various needs. Also, I don't think the package structure was documented or announced. Announced publicly? No, not that I know of. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module. Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
2010/11/2 Raymond Hettinger raymond.hettin...@gmail.com: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. from unitest.case import SkipTest. Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. See http://docs.python.org/whatsnew/2.7.html#updated-module-unittest -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 22:43, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:33, Nick Coghlanncogh...@gmail.com wrote: On Tue, Nov 2, 2010 at 12:35 PM, Brett Cannonbr...@python.org wrote: So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). Something to note in PEP 8, perhaps? If everyone agrees with making this policy, then yes. I'd like to reply a bit further, I'll do it as a reply to your earlier email though. Michael -Brett Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 3:58 PM, Guido van Rossum wrote: To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? I don't find anything offensive about it. The issues have to do with being able to find and analyze code. For example, to find-out what assert.ItemsEqual does, I have to figure-out that it was put in the case.py file. In Py2.6, you code use IDLE's Open Module tool to immediately bring up all the source for unittest. Then you could fire-up the class browser to quickly see and navigate the structure, but that also no longer works in Py2.7. Also, it used to be the just knowing the module name was sufficient to find the code with http://svn.python.org/view/python/branches/release26-maint/Lib/unittest.py?view=markup All you needed to study the code was a web browser and its find function. Now you need to open ten tabs to be able to browse this code. IOW, the packaging broke a read-the-source-luke style of research that I've been teaching people to use for years. I probably didn't articulate the above very well, but I think Martin said it more succinctly in this same thread. The other issue that Brett pointed out is that the file names now become part of the API, from unittest.utils import safe_repr. In the logging module, packaging was done well. The files fell along natural lines in the API, some of the components we usable separately and testable separately. Likewise with the xml packages. In contrast, the unittest module is full of cross-imports and tightly coupled pieces (like suite and case) have been separated. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 16:20 -0700, Raymond Hettinger a écrit : For example, to find-out what assert.ItemsEqual does, I have to figure-out that it was put in the case.py file. Well, it's a TestCase method, so it seems rather intuitive to look for it in case.py. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote: Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. That's a bummer. Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Le mardi 02 novembre 2010 à 16:32 -0700, Raymond Hettinger a écrit : On Nov 2, 2010, at 4:00 PM, Brett Cannon wrote: Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. That's a bummer. Sounds like a decision to split a module into a package is a big commitment. Each of the individual file names becomes a permanent part of the API. Even future additional splits are precluded because it might break someones dotted import (i.e. not a single function can be moved between those files -- once in unittest.utils, alway in unittest.utils). I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 23:00, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.) Much better times to discuss this would be when it was proposed or when it was done, not months after it has been shipped in a production release. [snip...] This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Broken imports between Python 2.7 code and any version of Python where unittest is re-merged back into a single module. I *know* that some people are using the new package structure directly, because the topic has come up on the Testing in Python mailing list. Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? Somewhat, yes. Screwing with unittest is always touchy as absolutely no one wants their tests to break, and that includes messing with imports. We could stick in import shims to shift everything into unittest/__init__.py, but the benefits you have outlined already don't strike me as not worth the hassle especially since you won't ever get your unittest.py format back. Absolutely, that would be the worst of all possible worlds - a monolithic huge module but still embedded in a package. People *are* using the existing package structure to import directly from (against my advice!) as this particular topic has been discussed on the Testing In Python mailing list. Although Raymond has been vociferous in his detestation of this particular split he does not represent a clear consensus in favour of merging back. Both Fred Drake and Barrry Warsaw voiced their approval and on the Clean up unittest API issue both yourself (Brett) and Antoine have supported keeping the existing structure. Alexander Belopolsky and Martin Loewis expressed difficulties with the new structure, but that was in response to the original email from Raymond that didn't seem (on my reading) to expressly suggest re-merging unittest back into a module but was instead seemed to be using it as an example. I am aware of the costs of having a package rather than single (however large it may be) module, but to my mind the benefits to maintenance still outweigh these cost. I'm happy to again discuss these benefits at great length, but having had the same conversation in person with Raymond twice and at repeated most of the points (but by no means all) in this thread on the mailing list it really feels like going round in circles. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses,
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 22:58, Guido van Rossum wrote: On Tue, Nov 2, 2010 at 3:47 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. The file splitting was done badly, so I don't think there any of the components are usable directly, i.e. from unitest.case import SkipTest. Also, I don't think the package structure was documented or announced. This is in contrast to the logging module which does have a clean separation of components and where it isn't unusual to import just part of the package. What is it you're seeing as a risk that I'm not seeing? Are we permanently locked into the exact ten filenames that are currently used: utils, suite, loader, case, result, main, signals, etc? Is the file structure now frozen? To spout a somewhat contrarian opinion, I just browsed the new unittest package, and the structure seems reasonable to me, even if its submodules are not particularly reusable. I've used this kind of style for development myself. What is so offensive about it? Amen. Although not that contrarian, others have spoken up in favour. The split is pretty obvious (in general - I'm sure its not perfect) and divided along major functionality. TestCase - case.py TestResult - result.py TestSuite - suite.py TextTestRunner - runner.py TestLoader - loader.py main function - main.py signal handling - signals.py The utils module is somewhat an odd one out as it is only used by case.py, but this is hardly the most egregious error in the world. If you can't guess where a class lives, __init__.py shows you explicitly (a clear advantage of exporting the public API at the top level ;-) Due to the original design of unittest (and I have many thoughts on that) the modules aren't strictly reusable (i.e. isolated from each other) - but many test frameworks (for example) just use the TestCase without using other components. I find it difficult to believe that this package structure is only acceptable if we make people import the TestCase from unittest.case and not expose it at the top level. As mentioned in another email, but this thread has many long and tedious emails, there is no clear consensus that there should be a remerger and I am aware that there are already some consumers of the new package structure. As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. All the best, Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. I don't think that anybody seriously expected the unittest package would be restructured again. The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. Personally I don't think it should -- it should by now be clear that this is not an area where one style will fit all. I also think that the convenience of one style over another might have something to do with the tools being used. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Nov 2, 2010, at 4:43 PM, Guido van Rossum wrote: The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. I was thinking of PEP 8 wording that listed the forces for and against. For example, ply.yacc and ply.lex was a very useful split (separately testable, natural division of concerns, no nested of cross-imports). The xml.sax, xml.dom, and xml.minidom was a nice split because it separated distinct tools. The xml packaging also worked well because it is easy to substitute in alternate parsers implementing the same API. I think we also want to recommend against putting much if any code in __init__.py. Some forces against packaging are that it breaks the class browser. As you say, different users of different toolsets are affected differently. For me, the unittest split broke my usual ways of finding out how the new methods were implemented. Another force against is what Brett pointed-out, that the package file structure becomes a permanent and unchangeable part of the API. It's a one-way street. In general, I think the advice should be that packaging should be done when there is some clear benefit beyond turning one big file into lots of smaller files. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 23:34, Michael Foord wrote: On 02/11/2010 23:00, Brett Cannon wrote: On Tue, Nov 2, 2010 at 15:47, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Nov 1, 2010, at 7:35 PM, Brett Cannon wrote: I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. I'm not sure I follow where we're stuck with the current package. AFAICT, the module is still used with import unittest. Yes, as far as you can tell, but who the hell knows what someone is doing with code you are *not* aware of. As I said, Python 2.7 shipped with the code structured like this, so it's possible someone is importing unittest.case.TestCase instead of unittest.TestCase. It is also shipped in unittest (and unittest2py3k I might add) so that users of earlier versions of Python can use the new features seamlessly. (unittest2 will be in Django 1.3.) unittest2 dammit. Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 5:03 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: Some forces against packaging are that it breaks the class browser. As you say, different users of different toolsets are affected differently. For me, the unittest split broke my usual ways of finding out how the new methods were implemented. Maybe the IDLE class browser can be fixed; there is plenty of code with this structure that can't or won't be restructured, no matter how strongly PEP 8 is worded. FWIW, personally I don't use the IDLE class browser -- I use Emacs, grep, and find. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
Antoine Pitrou solip...@pitrou.net writes: I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)? If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. -- \“Your [government] representative owes you, not his industry | `\ only, but his judgment; and he betrays, instead of serving you, | _o__)if he sacrifices it to your opinion.” —Edmund Burke, 1774 | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 12:47 am, ben+pyt...@benfinney.id.au wrote: Antoine Pitrou solip...@pitrou.net writes: I don't agree with this. Until it's documented, it's an implementation detail and should be able to change without notice. If it's an implementation detail, shouldn't it be named as one (i.e. with a leading underscore)? If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. And if that isn't the rule, then what the heck is? Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 8:47 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: I would say that names without a single leading underscore are part of the public API, whether documented or not. I don't recall this ever being the standard library's policy. There are many modules using leading underscores as hints, and many others which don't. Other people consider the presence of a docstring on a non-underscored name significant, while still others refer to the out-of-line documentation as definitive. For modules, an __all__ attribute is generally agreed on as definitive, if present, but that's a fairly limited case. At this point, there isn't a single clear way to determine if something is public API. I doubt it will be likely to agree on a single definition now without engendering a lengthy discussion on whether names can be changed to reflect such a policy (where backward compatibility is sure to be lost). I'm sticking to the out-of-line documentation to determine what's public. -Fred -- Fred L. Drake, Jr. fdrake at acm.org A storm broke loose in my mind. --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 16:43, Guido van Rossum gu...@python.org wrote: On Tue, Nov 2, 2010 at 4:39 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: As the maintainer of unittest I'd like to say that in the absence of clear consensus that the merger should happen, or a fiat from the BDFL, the merger won't happen. I believe that this is standard Python development process. I don't think that anybody seriously expected the unittest package would be restructured again. The remaining thrust of the thread seems to be whether PEP 8 should advise against breaking code up into many little modules. Personally I don't think it should -- it should by now be clear that this is not an area where one style will fit all. I also think that the convenience of one style over another might have something to do with the tools being used. This is not what I am suggesting for PEP 8. I want to say that a package's file structure should reflect the public API. I personally have no trouble with modules in packages that do not have a ton of objects in them. I just think if you have pkg/mod.py, pkg.mod should be exposed in the API, else name the file _mod.py. In the case of unittest that would just mean documenting that it's unittest.case.TestCase and that unittest.TestCase is for legacy reasons, much like os.path is just blindly added on to os even though it is a separate module(s). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 02/11/2010 02:35, Brett Cannon wrote: On Wed, Oct 27, 2010 at 03:42, Antoine Pitrousolip...@pitrou.net wrote: On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolskyalexander.belopol...@gmail.com wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward. [snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly. It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score. As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post. All the best, Michael Foord I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 19:50, Michael Foord fuzzy...@voidspace.org.uk wrote: On 02/11/2010 02:35, Brett Cannon wrote: On Wed, Oct 27, 2010 at 03:42, Antoine Pitrousolip...@pitrou.net wrote: On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolskyalexander.belopol...@gmail.com wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. Well the structure *does* match the API (which is primarily why I disagree with Raymond that this is a 'bad split'). But not the public API as documented, e.g., it's documented as unittest.TestCase, not unittest.case.TestCase as the file structure suggests. How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -Brett He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files Most of us? Raymond, Alexander and Martin are the only ones I *recall* complaining about the split specifically in this thread and not all of those were on the grounds you mention. Several people supported the split. Guido didn't object to it on these grounds and Antoine noted that finding core classes was generally straightforward. [snip...] So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. When designing packages from the ground up that is a sensible rule of thumb to follow, but usually follows naturally from good design. This doesn't necessarily mean that all the sub-modules will export public APIs for consumers, so this is where I disagree with this. Python's package system is a very useful way of providing internal structure for projects. That doesn't mean that this structure must always be exposed publicly. It should be as easy to navigate as possible (and there is plenty about the old unittest.py module that wasn't easy to navigate I can assure you), but I *don't* think that the new package fails in a substantially greater way on that score. As Guido points out, this may depend a lot on which tools you use. I could write more about the role and value of packages, I guess I'll save it for a blog post. All the best, Michael Foord I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, 03 Nov 2010 11:47:55 +1100 Ben Finney ben+pyt...@benfinney.id.au wrote: If someone wants to depend on some undocumented detail of the directory layout it's their problem (like people depending on bytecode and other stuff). I would say that names without a single leading underscore are part of the public API, whether documented or not. That's not what we are talking about; we are talking about their locations. If the official location is the unittest package, then I don't see why we should also support undocumented locations just because they happen to work. Otherwise we should also support e.g. unittest.unlink if the unittest package happens to have from os import unlink at its top. I don't think it's reasonable. Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 2 Nov 2010 19:57:48 -0700 Brett Cannon br...@python.org wrote: How could we have split the module into a package in a way that matched the API, whilst still retaining backwards compatibility with the old API? We had no choice but to export the public names at the top level. I'm not disagreeing with that. What I am saying is can now document that it's unittest.case.TestCase instead of having that just be an implementation detail. -1. unittest.TestCase is far simpler and more obvious that any javaesque qualified name. urllib.request and friends are already annoying enough. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Nov 2, 2010 at 7:50 PM, Brett Cannon br...@python.org wrote: This is not what I am suggesting for PEP 8. I want to say that a package's file structure should reflect the public API. But what does that mean? I could argue that unittest's structure (TestCase in case.py, etc.) reflects its public API just fine. I personally have no trouble with modules in packages that do not have a ton of objects in them. I just think if you have pkg/mod.py, pkg.mod should be exposed in the API, else name the file _mod.py. In the case of unittest that would just mean documenting that it's unittest.case.TestCase and that unittest.TestCase is for legacy reasons, much like os.path is just blindly added on to os even though it is a separate module(s). I really don't think we should encourage the use as unittest.case.TestCase -- it's unnecessarily introducing structure. I think it's fine now that the cat is out of the bag to document unittest.case.TestCase as an alternative spelling, but I don't think it should be the preferred one. os.path is so old that should not be taken as an example for anything. (It predates packages!) But it should not be changed either, there'd be too much churn. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Oct 27, 2010 at 03:42, Antoine Pitrou solip...@pitrou.net wrote: On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky alexander.belopol...@gmail.com wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. I realize I am a little late in this reply but issue 10273 linked to this and so now I am actually bothering to read this thread since it felt like bikeshedding when the thread began. I think the issue here is that the file structure of the code no longer matches the public API documented by unittest. Personally I, like most people it seems, prefer source files to be structured in a way to match the public API. In the case of unittest Michael didn't. He did ask python-dev if it was okay to do what he did, we all kept quiet, and now we have realized that most of us prefer to have files that mirror the API; lesson learned. But Python 2.7 shipped with this file layout so we have to stick with it lest we break any imports out there that use the package-like file structure Michael went with (which we could actually document and use if we wanted now that Michael has already broken things up). Reversing the trend by sticking all the code into unittest/__init__.py and then sticking import shims into the existing modules would be a stupid waste of time, especially considering the head maintainer of the package likes it the way it is. So basically it seems like we have learned a lesson: we prefer to have our code structured in files that match the public API. I think that is a legitimate design rule for the stdlib to follow from now on, but in the case of unittest it's too late to change it back (and it's a minor price to pay to learn this lesson and to have Michael maintaining unittest like he has been, plus we could consider using the new structure so that the public API matches the file structure when the need arises). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
While maintainers' convenience is a valid valid concern and some level of idiosyncrasy is healthy to allow active maintainers to code in their preferred style, I think users' convenience should come first when it conflicts with that of maintainers. Remember, code is written once and read many. This is particularly true about stdlib. A minor inconvenience of finding the right place to stick a new function in a large file does not in my view overweights a major inconvenience of not having all pieces in one place neatly organized in a linear order. I agree. While investigating an incompatibility in unittest2, I found that the breakage into multiple files makes it much harder to find out how things fit together, and where specifically a certain functionality is implemented. So join those who would have preferred this module to stay as a single 2000-line file. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 27, 2010, at 10:34 AM, R. David Murray wrote: To put your mind at ease, Barry, I'd not want to do that either :) Phew! But I wasn't worried, 'cause I know you're not insane. (Though the fact that you've effectively inherited the email package does bring that into question. :) But by (IMO good) design Generator, FeedParser, and Message are all supposed to be independent (use only each other's public APIs). And Header can be (and is, I think) used without the other pieces of email, as is true for other of the helper modules (base64mime, quoprimime, etc). Agreed. On the other hand, I have no clue why 'iterators.py' exists :) They're there for some corner use cases, but they do tend to be helpful. body_line_iterator() is probably the one I've used the most. The iterators are intended to be used independently. The one that bugs me most, though, is MIME. Combining all the mime stuff into one file seems like it would be a big win (not that it's possible, now). What is the benefit of email.mime.text.MIMEText over email.mime.MIMEText, when each of the files in the mime package consists of a single subclass? I think you're right that the extra level of module path is probably unnecessary. I'm not sure that means all the .py files in email/mime should be combined though. OTOH, `wc -l Lib/email/mime/*.py` is only 314 lines so I'm happy to defer to you on that. So, to clarify, like Raymond I'm not saying that packages are always bad. I'm just saying that packages are also not always good; and, further, that the number of lines of code in a file should, IMO, have nothing to do with the decision as to whether or not to create a package. +1 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 8:37 PM, Barry Warsaw wrote: If done well, a split can help improve the readability and discoverability of the code. No doubt that is true. The problem is that splitting can also impair discoverability. When unittest was in one file, you knew the filename was unittest.py from just the module name. Now, there are ten files and you have no way to guess any of their names or how the author grouped the functions and methods inside them. Let's say that I'm mystified about the difference between assertSameElements() and assertItemsEqual() and assertSetEqual(). So, I go to http://svn.python.org/view/python/branches/py3k/Lib/unittest/ . Now, how do you know which file contains the source? Formerly, I could call-up the one source file and do a find. Or, I could do OpenModule: unittest with IDLE and instantly see the source and let the class browser analyze the structure of the file. Now, that's not possible either. The source for those methods is now less discoverable. In the unittest/case.py file, the safe_repr() function function is called over 40 times and it is not used in any other file. So how do be benefit from it being defined in the utils.py file? ISTM, this saved nothing. The case.py file is over 1000 lines long and utils is 80. How did we benefit from that split? For me, it makes it harder to read the code because I have to go looking elsewhere for commonly called functions. The unittest module grew from one file in Py2.6 to ten files and one directory with 2500 lines in Py2.7. Was that a win? I've spent time trying to read through the changes and cannot follow it without having most of those ten files open in my editor. For me, it's a PITA to read the code. It doesn't help that the file split blew away the svn blame history, so I have a harder time being about to tell what changed. All that being said, I started this thread with: Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. The goal of the post was just to raise awareness that splitting modules is not a straight win. There are costs to discoverability and searchability. It makes pyclbr less useful. It makes svn viewer less useful. And it loses history. The split of unittest is a done deal. Who knows, it may have been a net win. Am just hoping that we all understand that it was not cost free. My hope is that the splitting modules unnecessarily does not become a trend. Packages should be the last tool we reach for, not the first. In many cases, flat is better than nested. Raymond P.S. I liked your qualifier, if done well. There's a lot in those three words. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 26 Oct 2010 22:06:37 -0400 Alexander Belopolsky alexander.belopol...@gmail.com wrote: While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. Are you saying that it has become a pile of medium-sized balls of mud? I would like to say thanks for the mud, Michael! It's high quality mud for sure. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Wed, Oct 27, 2010 at 2:33 PM, Ron Adam r...@ronadam.com wrote: I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server? Given the time frame until beta 1, I'd suggest leaving it in pydoc for now. We can look at possibly documenting it and moving it to http.servers for 3.3. (The patch is attached to issue 2001 for anyone else that wants to take a look at it) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 10/27/2010 08:51 AM, Nick Coghlan wrote: On Wed, Oct 27, 2010 at 2:33 PM, Ron Adamr...@ronadam.com wrote: I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server? Given the time frame until beta 1, I'd suggest leaving it in pydoc for now. We can look at possibly documenting it and moving it to http.servers for 3.3. (The patch is attached to issue 2001 for anyone else that wants to take a look at it) Cheers, Nick. Fantastic! Thanks Nick ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 26 Oct 2010 23:37:10 -0400, Barry Warsaw ba...@python.org wrote: On Oct 26, 2010, at 09:54 PM, Antoine Pitrou wrote: I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules. And that seems perfectly reasonable to me, especially if the alternative were that Michael was so frustrated with a massive single .py file that it discouraged him from working on it. If done well, a split can help improve the readability and discoverability of the code. I shudder at the madness that a single email.py file would cause. To put your mind at ease, Barry, I'd not want to do that either :) But by (IMO good) design Generator, FeedParser, and Message are all supposed to be independent (use only each other's public APIs). And Header can be (and is, I think) used without the other pieces of email, as is true for other of the helper modules (base64mime, quoprimime, etc). On the other hand, I have no clue why 'iterators.py' exists :) The one that bugs me most, though, is MIME. Combining all the mime stuff into one file seems like it would be a big win (not that it's possible, now). What is the benefit of email.mime.text.MIMEText over email.mime.MIMEText, when each of the files in the mime package consists of a single subclass? So, to clarify, like Raymond I'm not saying that packages are always bad. I'm just saying that packages are also not always good; and, further, that the number of lines of code in a file should, IMO, have nothing to do with the decision as to whether or not to create a package. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 9:31 AM, Guido van Rossum wrote: I would like Gregor Lingl's approval of turning turtle.py into a package. It might make some things harder for novices, e.g. trackebacks and just browsing the source code. Also many people don't expect to find any code in a file named __init__.py (and most of the time I agree with this). But the alternative isn't so great either, assuming we'll want strict backwards compatibility (I wouldn't want the instructions in Gregor's or anyone's book to start failing because of this). You can't rename turtle to turtle/turtle.py either, because then there'd be horrible confusion between turtle/ and turtle.py. IOW, yes, flat still seems better than nested here! Thanks for saying this. It might be a good time to also have a discussion about what may be an emerging trend of wanting to split standard library modules into packages. I understand the urge to split longer modules into multiple source files but would like to mention some of the advantages of keeping modules in a single source file. Taking the decimal module as an example: * Right now, I can pull up the source and entire history from subversion by knowing just the module name: http://svn.python.org/view/python/branches/py3k/Lib/decimal.py?view=markup , and I can search the entire module with nothing more sophisticated than find in the web browser. That no longer works with unittest. * I can easily email the file to someone or copy it back to a previous version of Python because only one file is involved. To look at the code, I don't have to guess how someone would have partitioned it into decimal.context.utils.threadlocal.py or somesuch. I don't need a packaging tool to distribute a module (pyparsing and BeautifulSoup are two examples of tools that are easily managed by just being in one file). * Some editors are set up to easily dance across multiple files and directories as if they were a single unit; however, many editors are not. Most file viewers and editors work better with single files.Take two operations in IDLE for example. If I press Alt-M for open module, I can retrieve all of the source for decimal (but this won't work with the logging or unittest packages). Once the source is visible, I press Alt-C to bring up a class browser for quickly reviewing the source (but this also won't work with packages). Several Python editors have this capability (provided by the pyclbr module) but they are defeated by packaging. * The Lib directory is more easily grepped when it is flat. The more we nest the Lib directory, the harder it will be to search the sources (while excluding the test directory). Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. I just wanted to bring up some of the advantages of having a single file for a library module. It would not be a good thing if some of the newer committers embarked on taking modules over 2000 lines and split them into packages. Right now, we have hundreds of source files, but it wouldn't take long for someone on a packaging mission to turn that into thousands. If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. Raymond___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: .. Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant.What were the benefits that justified unittest.py split? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettinger raymond.hettin...@gmail.com wrote: If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. +1 (or more) -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
2010/10/26 Alexander Belopolsky alexander.belopol...@gmail.com: On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: .. Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant. What were the benefits that justified unittest.py split? Mostly it was huge. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 12:18 PM, Benjamin Peterson wrote: 2010/10/26 Alexander Belopolsky alexander.belopol...@gmail.com: On Tue, Oct 26, 2010 at 1:39 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: .. Packaging is not always wrong. Maybe it was the right thing to do for unittest, maybe not. This is an example that I personally find ill-justified. Particularly annoying is the fact that opening __init__.py gives you a list of relative imports and sends you to the next file for everything else. Having both main.py and __main__.py seems redundant.What were the benefits that justified unittest.py split? Mostly it was huge. Now, it's huge and split across multiple files so it's harder to search, the class browser won't work, and the full source cannot be brought up immediately using just the module name. The svn annotations and history are munged-up. The components were highly interdependent so now every file has to start with a set of cross-imports. Well, at least the __init__.py is not full of code. That's something. FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 26 Oct 2010 12:34:30 -0700 Raymond Hettinger raymond.hettin...@gmail.com wrote: FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those. I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 26/10/2010 15:05, R. David Murray wrote: On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettingerraymond.hettin...@gmail.com wrote: If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. +1 (or more) Really, you prefer to work in single multi-thousand line modules than cleanly split smaller packages? (Is that how you develop your own projects, out of interest?) The cross imports are unfortunate (there are very few actually circular dependencies in the package though I think), but more of an indictment on the design of unittest than the decision to split it into a package. How the splitting happened is that I proposed it on this list many months ago and got one reply in favour and no dissenters. Benjamin actually did the splitting. I find unittest *massively* easier to maintain and navigate now that it is a package. It doesn't take very long to work out where the classes live (no time at all if you look in the module names - where the classes live is *very* obvious except for the TextTestResult which lives alongside the TestRunner). The package split into modules is into conceptual units, where the responsibility of the units is clear. This makes the code not only easier to navigate but easier to *think about*. The structure of unittest is now much clearer. I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. There is something to be said for consistency within the standard library, but I don't think it is a price worth paying in this particular case. All the best, Michael Foord -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Oct 26, 2010 at 4:46 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. Gadzooks, Michael! Something else we agree on. 2000-line modules are a source of maintenance pain. -Fred -- Fred L. Drake, Jr. fdrake at acm.org A storm broke loose in my mind. --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 10/26/2010 02:34 PM, Raymond Hettinger wrote: FWIW, it wasn't that big (approx 2500 lines). The argparse, difflib, doctest, pickletools, pydoc, tarfile modules are about the same size and the decimal module is even larger. Please don't split those. Sense you mention this... I've worked on pydoc to make it much nicer to use in a browser. While doing that I needed to reworked the server part. That resulted in a clean server thread object (and supporting parts) with no pydoc specific code in those parts. It can work as a stand alone module quite nicely. It's about 170 lines with around a third of that as documented examples that can also run as doctests. More to the point, it's a simple text/html server wrapped in a thread object. It can work as a starting point to using a browser as a user interface like pydoc does. There is a patch in the bug tracker, I just need to make some minor updates to it and it can go in, but I really need some code organizing/placement review help. I I'm wonder what you may think. Keep it in pydoc or move it to the HTTP package? Document it or not? Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 2:54 PM, Ron Adam wrote: I've worked on pydoc to make it much nicer to use in a browser. While you're at it. Can you please modernize the html and create a style sheet? Right now, all of formatting is deeply intertwined with content generation. Fixing that would be a *huge* improvement. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 10/26/2010 05:35 PM, Raymond Hettinger wrote: On Oct 26, 2010, at 2:54 PM, Ron Adam wrote: I've worked on pydoc to make it much nicer to use in a browser. While you're at it. Can you please modernize the html and create a style sheet? Right now, all of formatting is deeply intertwined with content generation. Fixing that would be a *huge* improvement. Half way there! The server will read one if it exists. ;-) I'd really like to get this part in before 3.2 beta, and then I'll add a basic style sheet and update the html code to use it for 3.3. The present patch fixes and updates all the functional parts and allows you to do every thing that you can do on the command line, but a LOT easier. I think You, Nick. or one of the other Core developers could probably have this finished up in an afternoon if you really wanted. All the parts work. It's more about checking and adjusting the packaging at this point. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, Oct 26, 2010 at 5:37 PM, Fred Drake fdr...@acm.org wrote: On Tue, Oct 26, 2010 at 4:46 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. Gadzooks, Michael! Something else we agree on. 2000-line modules are a source of maintenance pain. While I appreciate your and Michael's eloquence, I don't really see why five 400-line modules are necessarily easier to maintain than one 2000-line module. Splitting code into modules is certainly a good thing when the resulting modules can be used independently. This helps users write leaner programs, reduces mental footprint of individual modules, etc, etc. The split unittest module does not bring any such benefits. It still presents a single big-ball-of-mud namespace, only rather than implemented in a single file, it is now swept in from eight different files. If instead, unittest module was split into several top level modules so that test cases can be defined without pulling in test runner machinery and third party test runners could avoid importing irrelevant text runners, that would qualify as an improvement in my book, but probably not big enough to justify breaking compatibility. While maintainers' convenience is a valid valid concern and some level of idiosyncrasy is healthy to allow active maintainers to code in their preferred style, I think users' convenience should come first when it conflicts with that of maintainers. Remember, code is written once and read many. This is particularly true about stdlib. A minor inconvenience of finding the right place to stick a new function in a large file does not in my view overweights a major inconvenience of not having all pieces in one place neatly organized in a linear order. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 04:46 PM, Michael Foord wrote: I find the big-ball-of-mud style development, where everything lives inside huge monolithic modules, very painful. I also think that it is an extremely bad example for new developers. There is something to be said for consistency within the standard library, but I don't think it is a price worth paying in this particular case. +1 (or more). :) or-maybe-you'd-prefer-more-^L-y y'rs, -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Tue, 26 Oct 2010 16:46:15 -0400, Michael Foord wrote: On 26/10/2010 15:05, R. David Murray wrote: On Tue, 26 Oct 2010 10:39:19 -0700, Raymond Hettingerraymond.hettin...@gmail.com wrote: If someone wants to reorganize code for clarity, I would prefer keeping it within one file, bringing related functions together and using comment lines to mark the major sections. ISTM, this is cleaner than introducing a new directory and having multiple files that cross-import one another. +1 (or more) Really, you prefer to work in single multi-thousand line modules than cleanly split smaller packages? (Is that how you develop your own projects, out of interest?) Yes. Note that this is a more recent tendency, a few years ago I used to split things pretty finely right off the bat, but I found I wasted a lot of time on py-file interdependencies doing it that way. Now I generally only split when the pieces end up generalized enough that I want to reuse them. Which, granted, is often enough. But what I'm doing in that case is moving code out of a file at the top level of my libraryinto a new file at the top level of my library. Now, it's true that I haven't written any huge applications lately, but looking at a couple of typical projects the main files run between 1500 and 2000 lines (including comments), and I don't find those to be particularly big and certainly not awkward. I do understand the attraction of putting related stuff into separate files. I just think the balance may have tilted too far in the direction of nested and could stand to be nudged back toward flat :) Unittest is only 2600 lines total including comments. Typical file sizes are between 200 and 300 lines. Those feel tiny to me :) (And email is worse...a number of files only have a single class in them, and some of those classes consist of maybe 20 lines...) -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On Oct 26, 2010, at 09:54 PM, Antoine Pitrou wrote: I think it comes down to the preference of whoever works the most actively on it. Michael is the most active contributor to unittest by far, and I suppose he prefers it to be split into several submodules. And that seems perfectly reasonable to me, especially if the alternative were that Michael was so frustrated with a massive single .py file that it discouraged him from working on it. If done well, a split can help improve the readability and discoverability of the code. I shudder at the madness that a single email.py file would cause. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On breaking modules into packages Was: [issue10199] Move Demo/turtle under Lib/
On 10/26/2010 05:35 PM, Raymond Hettinger wrote: On Oct 26, 2010, at 2:54 PM, Ron Adam wrote: I wonder what you may think. Keep it in pydoc or move it to the HTTP package? Document it or not? I still would like to know what your thoughts are concerning where to put, and/or how to package, the simple threaded text server? Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
