Re: [Python-Dev] Hash collision security issue (now public)
On Wed, Jan 04, 2012 at 11:55:13AM +0100, Antoine Pitrou wrote: On Wed, 4 Jan 2012 09:59:15 +0200 Maciej Fijalkowski fij...@gmail.com wrote: Is it *really* a security issue? We knew all along that dicts are O(n^2) in worst case scenario, how is this suddenly a security problem? Because it has been shown to be exploitable for malicious purposes? I don't think that's news either. http://mail.python.org/pipermail/python-dev/2003-May/035907.html and http://twistedmatrix.com/pipermail/twisted-python/2003-June/004339.html for instance show that in 2003 it was clearly known to at least be likely to be an exploitable DoS in common code (a dict of HTTP headers or HTTP form keys). There was debate about whether it's the language's responsibility to mitigate the problem or if apps should use safer designs for handling untrusted input (e.g. limit the number of keys input is allowed to create, or use something other than dicts), and debate about just how practical an effective exploit would be. But I think it was understood to be a real concern 8 years ago, so not exactly sudden. Just because it's old news doesn't make it not a security problem, of course. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
Stephen J. Turnbull wrote: Andrew Bennetts writes: No, that just means you shouldn't trust *root*. Which is where a VM is a very useful tool. You can have the “as root” environment for your tests without the need to have anything important trust it. Cameron acknowledges that he missed that. So maybe he was right for the wrong reason; he's still right. But in the current context, it is not an argument for not worrying, because there is no evidence at all that the OP set up his buildbot in a secure sandbox. As I read his followups, he simply didn't bother to set up an unprivileged user and run the 'bot as that user. I made no claim about how the bot was deployed. The point I was disputing was more general than how one specific bot is deployed. To quote the mail I was replying to again: “HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted.” This sentiment was expressed strongly and repeatedly in several mails. It was this overly broad assertion I was addressing, and happily my argument was apparently convincing. I'm fine with “It's not worth running the tests as root because the overhead of making a secure setup for it with a VM etc is too hard with our very limited volunteer resources.” I'm not fine with “We mustn't run them as root because it's impossible to do it safely.” That's all I'm saying. […] that was *not* the case; the assumption is falsified. Nevertheless, several people who I would have thought would know better are *all* arguing from the assumption that the OP configured his test system with security (rather than convenience) in mind, and are castigating Cameron for *not* making that same assumption. To my mind, every post is increasing justification for his unease. :-( I certainly hope I wasn't so severe as to be castigating! If I was Cameron has been kind enough to not show any offense. And that's why this thread belongs on this list, rather than on Bruce Schneier's blog. It's very easy these days to set up a basic personal VM, and folk of goodwill will do so to help the project with buildbots to provide platform coverage in testing new code. But this contribution involves certain risks (however low probability, some Very Bad Things *could* happen). Contributors should get help in evaluating the potential threats and corresponding risks, and in proper configuration. Not assurances that nothing will go wrong because you probably run the 'bot in a VM. For the record, in case it isn't obvious, I think a buildslave that runs the tests as root that doesn't take precautions like using a VM dedicated to just running the tests (and not running the buildslave) is a bad idea. Although given that there's a very limited supply of volunteer labour involved in configuring and administering buildslaves I'm not surprised to hear this has happened. :( I don't object at all to folks like Cameron asking questions to ensure that these systems are secure enough. I think that's a good thing! I don't even object to treating someone saying “run as root” as a red flag requiring further explanation. What I was objecting to was an apparent willingness to make an unnecessary compromise on software quality. I care about the security of contributors' buildslaves. I also care about the reliability of Python. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (3.2): Issue #11956: Skip test_import.test_unwritable_directory on FreeBSD when run as
On Fri, Oct 07, 2011 at 08:27:01AM +1100, Cameron Simpson wrote: […] | running buildbot tests as root does not reflect the experience of | non-root users. It seems some tests need to be run both ways just for | correctness testing. | | (except I'd say all, not some) No. Terry is right and you are ... not. Most tests need no special privileges - they're testing language/library semantics that do not depend on the system facilities much, and when they do they should work for unprivileged users. You could also say that most tests that work on Linux work on FreeBSD too, so when they work on Linux they should work for FreeBSD too… so why bother running tests on FreeBSD at all? The reason is because the assumptions behind that “should” are wrong frequently enough to make it worth running tests in both environments. Like Glyph, I think that “running as root” is sufficiently different environment to “running as typical user” (in terms of how POSIX-like systems behave w.r.t. to things like permissions checks) to make it worthwhile to regularly run the whole test suite as root. HOWEVER, the whole suite should not be _tested_ as root because the code being testing is by definition untrusted. No, that just means you shouldn't trust *root*. Which is where a VM is a very useful tool. You can have the “as root” environment for your tests without the need to have anything important trust it. […] Root _is_ special, within the host and with scope to misbehave beyond the host. 1: The permission system does _not_ behave the same for root as for other users. Those are arguments *for* running tests as root! 2: Root _can_ corrupt things anywhere in the system (within the VM, of course, but the builtbot is a subset of it). A normal unprivileged user This appears to be a key error in your logic. There's no fundamental reason why “tests run as root inside a VM” must necessarily imply “buildbot process is run inside that same VM and is therefore vulnerable to code in that test run.” It may be more convenient to deploy it that way, but I'm sure it's possible to have a buildslave configured to e.g. start a pristine VM (from a snapshot with all the necessary build dependencies installed) and via SSH copy the the source into it, build it, run it, and report the results. The VM could be fully isolated from the real network and filesystem etc if you like. Given that it is certainly possible to run tests as root about as securely as running them without root, do you still feel it is not worth running the tests as root? The prudent way to run the buildbots, especially if they cycle (refetch newer codebase, rebuilt, retest) instead of (scrub VM, reinstall, install built system, etc) is: - a user to fetch source and dispatch builds - possibly a distinct user to run the builds - definitely a distinct user to run the test suite If we're talking prudence, then surely s/user/VM/ is even better :) -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comments of the PEP 3151
Ethan Furman wrote: […] or EINTRError in my order of preference. Please not that last one! ;) Why not, exactly? When EINTR happens it's frequently a surprise, but programmers new to the concept can always search the web for advice on what causes it and how to deal with it (and after several attempts at dealing with it they may even get it right). Searching Google for “InterruptedError” isn't going to find anything helpful today, and eventually what I expect it would find is a bunch of pages saying “Look up EINTR.” How about we just cut out that middle step and call it what the rest of the world calls it? If “InterruptedError” were going to be used for anything other than EINTR then I could see an argument for abstracting the concept behind a platform-independent name. But I think it would be a mistake to treat anything else as being the same as EINTR. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 11: Dropping support for ten year old systems
Martin v. Löwis wrote: [...] Ubuntu: http://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Version_timeline (http://www.ubuntu.com/products/ubuntu/release-cycle seems to be down) I'd prefer something more official than Wikipedia, though. https://wiki.ubuntu.com/Releases -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] doctest, unicode repr, and 2to3
Martin v. Löwis wrote: [...] Any proposal appreciated. I propose screaming “help me, I have written a test suite using nothing but string matching assertions, what is wrong with me?!” -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7 alpha 2
Martin v. Löwis wrote: [...] But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. How so? If they use anything that is new in 2.8, they *will* need to drop support for anything before it, no??? I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Most likely, Twisted supports 2.8 *today* (hopefully). But how does that help Twisted in moving to 3.2? I'm not talking about Twisted moving to 3.x (FWIW, I think the only movement there so far is some patches for some -3 warnings). The situation I'm describing is a project X that: (a) has 2.x-only dependencies, and (b) would like to be as close as possible to 3.x (because they like the new features and/or want to be as ready as possible to jump when (a) is fixed). So just because project X depends on e.g. Twisted, and that Twisted in turn still supports 2.4, doesn't mean that X cannot move to 2.8, and doesn't mean it would get no benefit from doing so. [...] No, it won't. It might be if people move to 2.8 *and* drop 2.5, but they likely won't. But this is my point. I think they would as an intermediate step to jumping to 3.x (which also requires dropping 2.5, after all!), if for some reason they cannot yet jump to 3.x, such as a 2.x-only dependency. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7 alpha 2
Martin v. Löwis wrote: [...] I've done a fair bit of 3.x porting, and I'm firmly convinced that 2.x can do nothing: [...] Inherently, 2.8 can't improve on that. I agree that there are limitations like the ones you've listed, but I disagree with your conclusion. Maybe you assume that it's just as hard to move to 2.8 (using the py3k backports not available in say 2.5) as it is to 3.x? But a hypothetical 2.8 would also give people a way to move closer to py3k without giving up on using all their 2.x-only dependencies. I think it's much more likely that libraries like Twisted can support 2.8 in the near future than 3.x. Then, when all of your dependencies (or viable alternatives to those dependencies) are available for 3.x, you'll have an easier transition if you can start from a 2.x with fewer differences in features. Fundamentally the more 2.x can converge on 3.x, the easier it will be for users to make the leap, because it will be a smaller leap. The longer the 2.x series lives, the more these newer 2.x versions like 2.7 and maybe even 2.8 will be available on common platforms for people to depend upon as minimum versions, which means that as time goes by they can depend on a version that's closer to 3.x. And so again, the leap becomes easier to make. So to me it's pretty clear that 2.8 *can* improve the transition path to 3.x. It may or may not be a worthwhile use of effort for python-dev to make 2.8, but that's different to saying it's inherently pointless. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Easy way to detect filesystem case-sensitivity?
Antoine Pitrou wrote: Robert Kern robert.kern at gmail.com writes: Since one may have more than one filesystem side-by-side, this can't be just be a system-wide boolean somewhere. One would have to query the target directory for this information. I am not aware of the existence of code that does such a query, though. Or you can just be practical and test for it. Create a file foobar and see if you can open FOOBAR in read mode... Agreed. That is how Bazaar's test suite detects this, and it works well. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BZR mirror and pushing to Launchpad
R. David Murray wrote: I just posted a (tiny) patch to the tracker, and for the exercise of it I thought I would push the branch out to Launchpad as suggested in the wiki (http://wiki.python.org/moin/Bazaar). It looks like it is uploading every file in the branch instead of the delta from the trunk. Did I do something wrong in my local bzr setup, or is this the expected behavior? It should Just Work, i.e. pushing that branch to Launchpad should stack automatically. I just spoke to one of the code.launchpad.net developers and everything seems to be working normally on his end. Can you provide a bit more information about exactly what you did, so we can try to reproduce the problem? Also, as this may be more of an issue with Launchpad than the Python bzr import, it might be better to continue this conversation on launchpad-users (subscribe at https://launchpad.net/~launchpad-users) rather than python-dev. I guess it depends on whether or not python-dev would consider this noise or not... -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BZR mirror and pushing to Launchpad
Ross Light wrote: Yes, this is the expected behavior. Bazaar will upload all of the revisions since it is not stacking off of another branch. You could try using the Launchpad or Python.org mirrors as a stacking branch, as described here: http://doc.bazaar-vcs.org/bzr.dev/en/user-guide/index.html#using-stacked-branches Actually, the Launchpad code server is configured to make bzr automatically stack on the appropriate branch. The docs you point to there are about manually specifying the branch to stack on. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BZR mirror and pushing to Launchpad
Ross Light wrote: [...] However, my understanding is that Launchpad will not automatically stack if you are using the Python.org branch; you must use the Launchpad mirror. If you want to stack off the Python.org branch, then I think you need to manually stack. It is true that if the source branch that you push to Launchpad is not in a stacking-capable format, then the automatic stacking will not happen. But that shouldn't be the case with the Python.org branch; http://code.python.org/python/trunk/ uses the latest format which is stacking-capable. On the other hand, it does appear that https://code.launchpad.net/~rdmurray/python/bug5450 is a branch that is not stacking-capable. I'm not sure how this could happen by following the instructions on the the wiki page. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Missing operator.call
Hrvoje Niksic wrote:
Is there a reason why the operator module doesn't have an operator.call
function?
Python 2.6 adds operator.methodcaller. So you could use
operator.methodcaller('__call__'), but that's not really any better than
lambda x: x().
A patch to add operator.caller(*args, **kwargs) may be a good idea. Your
example would then be:
map(operator.caller(), lst)
That reads ok to me. I think this reads better though:
[callable() for callable in lst]
wink
-Andrew.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Partial function application 'from the right'
Mike Klaas wrote: On 29-Jan-09, at 3:21 PM, Daniel Stutzbach wrote: [...] The meaning which numpy attributes to Ellipsis is also the meaning that mathematical notation has attached to Ellipsis for a very long time. And yet, python isn't confined to mathematical notation. *, ** are both overloaded for use in argument lists to no-one's peril, AFAICT. With the crucial difference that * and ** are purely syntax, but Ellipsis is an object. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess crossplatformness and async communication
Daniel Stutzbach wrote: [...] If you really need to communicate with multiple subprocesses (which so far has not been suggested as a motivating example), then you can use select(). Not portably. select() on windows only works on sockets. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] I would like an svn account
Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jan 3, 2009, at 11:54 AM, Ulrich Eckhardt wrote: 1. I think that a patch can not e.g. capture a moved, renamed or deleted file. Further, it can not handle e.g. things like the executable bit or similar things that SVN otherwise does manage. That is what makes a patch only partially suitable. Bazaar bundles handle moved, renamed and deleted files and directories, afaik. Executable bits and other file metadata are harder because such properties are OS specific, and Bazaar tries to be OS agnostic. Actually, Bazaar also handles the execute bit too. But it doesn't try to capture any other file metadata like mtime or permissions. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
s...@pobox.com wrote: Steve Unfortunately there are doubtless programs out there that do rely Steve on actions being taken at shutdown. Indeed. I believe any code which calls atexit.register. Steve Maybe os.exit() could be more widely advertised, though ... That would be os._exit(). Calling it avoids calls to exit functions registered with atexit.register(). I believe it is both safe, and reasonable programming practice for modules to register exit functions. Both the logging and multiprocessing modules call it. It's incumbent on the application programmer to know these details of the modules the app uses (perhaps indirectly) to know whether or not it's safe/wise to call os._exit(). You could call sys.exitfunc() just before os._exit(). -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r66863 - python/trunk/Modules/posixmodule.c
Martin v. Löwis wrote: [...] There is a certain prevention already that later maintenance fixes don't break the earlier ones: those fixes typically get checked into the trunk also, where the tests do exist. So the committer would find out even before the patch gets to the maintenance branch. By this logic, the maintenance branch would need no tests at all, because they are all in trunk already! -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Consolidating names and classes in the `unittest`module (updated 2008-07-15)
Nick Coghlan wrote: [...] What did you think of the check idea at the end of the email? Test assertions: check(x).almost_equal(y) check(x).is_(y) check(x).in_(y) check(x).equals(y) Test negative assertions: check(x).not_almost_equal(y) check(x).is_not(y) check(x).not_in(y) check(x).not_equal(y) Wow. This is a textbook example of a bikeshed discussion. The names (and now syntax!) of assertions are the most cosmetic issue there is with the unittest module, yet I see over *200* messages about it. Deeper, but important issues, such as how to raise structured exceptions that a GUI test runner could usefully display and introspect, are ignored. I can list lots of others. For assertions, just flip a coin already (or a roll a d20, perhaps...). Can we perhaps devote this considerable energy to significant improvements, please? -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest PEP do's and don'ts (BDFL pronouncement)
Michael Foord wrote:
Raymond Hettinger wrote:
[...]
If some people want to proceed down the path of useful additions,
I challenge them to think bigger. Give me some test methods that
improve my life. Don't give me thirty ways to spell something I can
already do.
I assert that... the following changes do meet those conditions:
assertRaisesWithMessage - for testing the error messages from library
functions, where the error message is part of the API under test (I'm
less convinced with the need for a regex matching version myself.)
This one is easily solved by making assertRaises return the exception it caught.
e.g.:
exc = self.assertRaises(AttributeError, getattr, foo, 'bar')
self.assertEqual('Foo' object has no attribute 'bar', exc.message)
At least Twisted and bzr have already made this exact change. It requires no
new
methods, and is more flexible than the proposed assertRaisesWithMessage.
-Andrew.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest PEP do's and don'ts (BDFL pronouncement)
Ben Finney wrote: Andrew Bennetts [EMAIL PROTECTED] writes: This one is easily solved by making assertRaises return the exception it caught. That breaks one simple feature of the unittest API: that all the test methods will either raise a failure asertion, or return None. How is returning None a feature? I've never seen code that somehow depends on assertRaises returning None. This “feature” is not documented as being significant in the unittest module documentation anywhere. It is not mentioned anywhere in the *eight* pages of the xUnit Test Patterns book[1] dedicated to Assertion Methods in their general form. Where did you get the notion that it is a feature? Further, I have lots of evidence that in practice returning the exception instance from assertRaises is not a problem, and is in fact quite useful. I'd quote “Practicality beats purity”, but I'm not even sure if it is purity that you have in mind. Demanding that assertion methods return None seems like foolish consistency and dogma. -Andrew. [1] _xUnit Test Patterns: Refactoring Test Code_, by Gerard Meszaros. http://xunitpatterns.com/. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest PEP do's and don'ts (BDFL pronouncement)
Ben Finney wrote: Andrew Bennetts [EMAIL PROTECTED] writes: [...] How is returning None a feature? A test method having exactly one meaning is a feature. If it's consistent across the API, the API retains a level of simplicity. Your reply makes no sense to me. I am proposing that it should have exactly one meaning. Callers will be free to ignore the return value if they don't need it, and will see zero difference in behaviour. It seems you have a different meaning of “meaning” than I do, which suggests that this conversation is doomed. I hope I don't sound hostile, I'm just bewildered. Your argument isn't making any sense to me. I can of course keep using TestCase subclasses that override assertRaises to be more useful. And I will, because there are no actual downsides that I've seen any evidence of. I'd be even happier if the standard library adopted this improvment, though. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unittest PEP do's and don'ts (BDFL pronouncement)
Ben Finney wrote: [...] I hope that clarifies it. The name of a thing, in Python especially, is very important; in an API, even more so. If the behaviour of the function isn't matched by the name, it's a poorly chosen name, a poorly designed function, or both. It doesn't really clarify it for me. We're both just repeating ourselves, though. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Consolidating names and classes in the `unittest` module (updated 2008-07-15)
Ben Finney wrote: Stephen J. Turnbull [EMAIL PROTECTED] writes: Ben Finney writes: Removal of ``assert*`` names There is no overwhelming consensus on whether to remove the ``assert*`` names or the ``fail*`` names; 7 to 1 is overwhelming in my book. See Message-ID: [EMAIL PROTECTED] From: Antoine Pitrou [EMAIL PROTECTED] That measured only usage of unittest *within the Python standard library*. Is that the only body of unittest-using code we need consider? Three more data points then: bzr: 13228 assert* vs. 770 fail*. Twisted: 6149 assert* vs. 1666 fail*. paramiko: 431 assert* vs. 4 fail*. The data seems pretty overwhelmingly in favour of keeping assert*. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses
Ben Finney wrote: Guido van Rossum [EMAIL PROTECTED] writes: [...] I like using only the assertKeyword variants, removing assert_, fail*, and assertEquals. I'm the opposite. I prefer the 'fail*' variants over the 'assert*' variants, because fail tells me exactly what the function will *do*, while assert leaves it implicit what will actually happen if the assertion is false. For this reason, I'd prefer the fail* names to be recommended, and the assert* names to be, if not deprecated, then at least second-class citizens. On the other hand, “assert*” names are positive statements of what the behaviour of the system-under-test is. And conversely, “fail*” names are a bit like implementation details of how the test is written. So I personally have a mild preference for the assert* names. My suspicion is that it will be very hard to get consensus on the colour of this particular bike shed :) -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py3k DeprecationWarning in stdlib
Brett Cannon wrote: On Wed, Jun 25, 2008 at 6:08 AM, Andrew Bennetts [...] Should I file a bug for this? If you want, but Benjamin plans to undocument this for users along with all other test.support stuff (which I agree with). Most of the APIs in test.support were just quickly written and have not necessarily been thought through in order to make sure that the APIs makes sense (like in this case). Ok, then we're back to there being no supported way to write tests that need to intercept warnings. Twisted has already suffered from this (JP reports that Twisted's assertWarns is broken in 2.6), and I doubt it's alone. So I guess I am filing a bug after all... :) -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py3k DeprecationWarning in stdlib
Nick Coghlan wrote: [...] I forgot this had already been added to the Python regression test machinery, so it will just be a matter of updating the relevant tests to use it: That's a nice surprise! I'm glad the standard library is growing facilities like this. I think it could be improved a little, though: http://docs.python.org/dev/library/test.html#module-test.test_support test.test_support.catch_warning(record=True)¶ Return a context manager that guards the warnings filter from being permanently changed and records the data of the last warning that has been issued. The record argument specifies whether any raised warnings are captured by the object returned by warnings.catch_warning() or allowed to propagate as normal. The description doesn't really make the record=False case clear. This context manager is doing two different jobs: 1) restore the filters list and showwarning function to their original state when done, and 2) optionally (if record=True) record the last warning in the as target. That feels a bit weird. I think a clearer way to provide that functionality would be with two separate context managers: one that copies and finally restores the filters list and showwarnning function called protect_warnings_module, and then catch_warnings to record the warnings. The catch_warnings context manager could reuse the protect_warnings_module one. with protect_warnings_module: seems easier to understand (and document) than with catch_warning(record=False). Should I file a bug for this? The context manager is typically used like this: with catch_warning() as w: warnings.warn(foo) assert str(w.message) == foo Maybe this is a YAGNI, but what if an expression triggers multiple warnings? Perhaps the WarningMessage object ought to keep a list rather than just the last warning. Another thought: if the warnings module had a public, documented way to manipulate the filter list (e.g. warnings.get_all_filters() and warnings.set_all_filters(...)), then people could build these sorts of test facilities for themselves if the test_support one turns out to be too limiting or otherwise a poor fit. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Lazy module imports and post import hook
Neal Becker wrote: Christian Heimes wrote: I've attached the first public draft of my first PEP. A working patch against the py3k branch is available at http://bugs.python.org/issue1576 Christian Note also that mercurial has demandimport http://www.selenic.com/mercurial/wiki/ And that bzr has lazy_import (inspired by mercurial's demandimport): http://codebrowse.launchpad.net/~bzr/bzr/trunk/annotate/3170?file_id=lazy_import.py-20060910203832-f77c54gf3n232za0-1 http://bazaar-vcs.org/ -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).
Guido van Rossum wrote: On Dec 11, 2007 4:54 PM, Jan Claeys [EMAIL PROTECTED] wrote: Op vrijdag 07-12-2007 om 07:26 uur [tijdzone -0700], schreef Sean Reifschneider: I would say that this is an optimization that helps a specific set of platforms, including one that I think we really care about, the OLPC which needs it for decreased battery use. Almost every laptop user would benefit from it, and even some desktop or server users might save on their electric power bill... Do you have data to support this claim? http://www.lesswatts.org/projects/powertop/powertop.php Some quotes plucked from that page: “In the screenshot, the laptop isn't doing very well. Most of the time the processor is in C2, and then only for an average of 4.4 milliseconds at a time. If the laptop spent most of its time in C4 for at least 20 milliseconds, the battery life would have been approximately one hour longer.” “When running a full GNOME desktop, 3 wakeups per second is achievable.” There's considerable effort being invested in the GNOME and Linux software stack at the moment to get rid of unnecessary CPU wakeups, and people are reporting significant improvements in laptop power consumption as a result of that work. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Segfault
Maciej Fijalkowski wrote: IMHO this shouldn't segfault: import thread while 1: f = open(/tmp/dupa, w) thread.start_new_thread(f.close, ()) f.close() while it does on cpython 2.5.1 , linux box. May I consider this a bug? Yes, that's a bug. Please file it at http://sourceforge.net/bugs/?group_id=5470. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] make iter() return an empty iterator?
Georg Brandl wrote: Sure, you could use ``iter(())`` or ``iter([])``, but for consistency's sake wouldn't it make sense for ``iter()`` to return an empty iterator, as ``str()`` returns an empty string etc.? I had no idea that str() or int() would do that. file() certainly doesn't! :) I don't really think there's much reason to make iter() work. As you say, iter([]) works just fine. For those rare times you want an empty iterator, I don't think the two extra characters is much of a price to pay. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.rename on windows
Raghuram Devarakonda wrote: Hi, I have submitted a patch (http://www.python.org/sf/1704547) that allows os.rename to replace the destination file if it exists, on windows. As part of discussion in the tracker, Martin suggested that python-dev should discuss the change. Does MOVEFILE_REPLACE_EXISTING mean the rename over an existing file is actually atomic? I cannot find any MSDN docs that say so (and I've seen some that suggest to me that it probably isn't). If it's not atomic, then this doesn't offer any advantage over shutil.move that I can see (and in fact would damage the usefulness of os.rename, which is currently atomic on all platforms AFAIK, even though it cannot succeed all the time). If MOVEFILE_REPLACE_EXISTING miraculously turns out to be atomic, then my opinion is: * this feature would be very useful, and I would like a cross-platform way to do this in Python. * this feature should not be called os.rename, which for years has done something else on some platforms, and so changing it will invite unnecessary breakage. A new function would be better, or perhaps an optional flag. I propose os.atomic_rename. Also, I assume this cannot replace files that are in use? Currently, os.rename() on windows uses the API MoveFile() which fails if the destination file exists. The patch replaces this API with MoveFileEx() and uses the flag MOVEFILE_REPLACE_EXISTING which causes the destination file to be replaced if it exists. However, this change is subtle and if there is any existing code that depends on current os.rename behaviour on windows, their code is silently broken with (any) destination file being overwritten. But the functionality of replacing is important and I would like to know the best of way of supporting it. If it is deemed that this change is not good to go in as-is, how about having an optional parameter to os.rename (say, win_replace) that can be used by callers to explicitly request replacing? I'd be ok with a flag, but it should have a cross-platform name. require_atomic or replace or something like that. I think a new function makes more sense, though. I must also point out that the patch uses another flag MOVEFILE_COPY_ALLOWED effectively allowing renamed files to be on separate file systems. The renaming in this case is not atomic and I used this flag only to support current functionality. It is not a bad idea to disallow such renames which brings it in line with the behaviour on many unix flavors. This also has the potential to break code but not silently. I don't quite follow what you're saying here, but I'd be against an operation called rename that sometimes was atomic and sometimes wasn't. Lastly, I found an old discussion about the same topic by this list. http://mail.python.org/pipermail/python-dev/2001-May/014957.html Even though Guido indicated that he doesn't support API change in this thread, I am posting again as I did not see any one mention MoveFileEx() in that thread. Does MoveFileEx solve the atomicity problem that Guido raised in that thread? I don't think it does, so I think the situation is still the same. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SoC proposal: multimedia library
Lino Mastrodomenico wrote: Hello everyone, I would like to participate as a student in google Summer of Code and I'm interested in feedback on a multimedia library for Python. The library I propose should have the following features: * the capability to extract and decompress video and audio from a few common multimedia file format; * and, vice versa, it can create a compressed video file from a series of still images and from uncompressed audio; * it should have an interface as simple and pythonic as possible; * it must be cross-platform, easy to extend and not be limited to a single file container format or to some specific audio or video compression format; * at least a subset of the supported formats should be available without any external dependency. Except I guess for the last point, don't the Python bindings for GStreamer already provide this? -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitext('.cshrc')
Hi Martin, Martin v. Löwis wrote: [EMAIL PROTECTED] schrieb: [...] The use-cases being discussed here would be better served by having new APIs that do particular things and don't change existing semantics, though. For example, a guess_mime_type(path) function which could examine a path and figure out its mime type based on its extension (among other things). I disagree. The current behavior is incorrect (and always has been); the new behavior is clearly correct. Glyph's proposing that rather than risk breaking existing code (and in the worst possible way: silently, giving wrong answers rather than exceptions), we examine what benefits changing splitext would bring, and see if there's a way to get those benefits without that risk. I don't think that's an idea to dismiss out of hand. You silently omitted this part of glyph's objection: [EMAIL PROTECTED] wrote: [...] Incompatible changes may be *acceptable* for feature releases, but that doesn't mean they are desirable. The cost of incompatibility should be considered for every change. This cost is particularly bad when the incompatibility is of the silent breakage variety - the change being discussed here would not be the sort of thing that one could, say, produce a warning about or gently deprecate; anything relying on the old behavior would suddenly be incorrect, and any software wishing to straddle the 2.5-2.6 boundary would be better off just implementing their own splitext() than relying on the stdlib. At the moment you seem to be saying that because you're possibly fixing some currently buggy code, it's ok to possibly break some currently working code. I'm not sure that's a good idea, and I expect it will create headaches for someone trying to be compatible with both 2.5 and 2.6. If the existing behaviour was clearly buggy by the existing documentation, rather than undefined, this would be less of a concern. The point about silent failures is an important one too. This will cause some code that was working correctly to give wrong answers, rather than an exception or any indication that this previously correct use is now incorrect and causing bugs. If splitext was removed and replaced with, say, a split_ext that behaves as you (and basically everyone, myself included) agrees it should behave, then code won't silently do the wrong thing. It will immediately and visibly fail, in an understandable way. Similarly, if split_ext was added side-by-side with splitext, and splitext changed to emit DeprecationWarnings for a release, there's minimal risk that existing code that works would break. See also the comment at the bottom of http://sourcefrog.net/weblog/software/aesthetics/interface-levels.html about perverse interface design. I wonder if First, do no harm should be an explicit guideline when it comes evaluating incompatible changes, even if they fix other things? There seems to be the problem that all existing users of splitext that may encounter '.foo' will need to audit their code to see if they are relying on previously undefined behaviour that will change. I think it would be much more helpful for python to somehow help users discover this fact, than simply define the previously undefined behaviour to be different to what they might have been relying on. All that said, I don't think I've ever used splitext myself, so I don't care so much about what happens to it. But I do worry about the general approach to backwards compatibility. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] splitext('.cshrc')
Josiah Carlson wrote:
[...]
Offer a new splitext that uses X on posix and Y on win32, but causes a
DeprecationWarning with pointers to the two renamed functions that are
available on both platforms.
For people who want the old platform-specific functionality in previous
and subsequent Pythons, offer the ability to disable the
DeprecationWarning via a module global; at least until the function is
removed in some future Python, at which point path.splitext would cause
an AttributeError .
It's not about cross-platform behaviour. It's already the same:
import posixpath, ntpath
ntpath.splitext('.cshrc')
('', '.cshrc')
posixpath.splitext('.cshrc')
('', '.cshrc')
And as I understand it, the current proposal would change this behaviour the
same way on win32 and POSIX, although I may not have kept up with that part of
the discussion enough. Is there a genuine desire for this to behave differently
on different platforms?
I think what I'd probably like to see is the existing undefined (but
well-understood and predictable) behaviour to be declared defined, and a warning
put in the documentation that it probably doesn't do what you expect for this
case. And then for those that want the new behaviour, provide a new function
(and prominently advertise it in the relevant part of the documentation). Yes
it sucks having two similar functions, which will inevitably have similar names,
but that's the price you pay for getting the interface wrong the first time
around.
-Andrew.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python and the Linux Standard Base (LSB)
On Fri, Dec 01, 2006 at 12:42:42AM +0100, Jan Claeys wrote: Op donderdag 30-11-2006 om 21:48 uur [tijdzone +], schreef Steve Holden: I think the point is that some distros (Debian is the one that springs to mind most readily, but I'm not a distro archivist) require a separate install for distutils even though it's been a part of the standard *Python* distro since 2.3 (2.2?) So, it isn't that you can't get distutils, it's that you have to take an extra step over and above installing Python. No, it just means that several parts of the python.org source package are spread over several binary packages, just like happens with hundreds or thousands of other packages, and any Debian (or Ubuntu or other distro doing this) administrator worth his or her money should be aware of that, and be able to find those packages. In both the current Debian and Ubuntu releases, the python2.4 binary package includes distutils. See for yourself at http://packages.debian.org/cgi-bin/search_contents.pl?searchmode=filelistword=python2.4version=stablearch=i386page=1number=all if you like. So I'm not sure what the fuss is about. Maybe python.org can include several logical divisions in the python.org distribution and make it easy for OS distro packagers to make separate packages if they want to, as most of them are quite happy to have less work to do, provided the upstream divisions do more or less what they want. ;-) (Oh, and such a division should IMHO also include a minimal python for embedded/low-resource hardware use, where things like distutils, GUI toolkits, a colelction of 20 XML libraries and documentation are most likely not needed.) There's already a python2.4-minimal package in Debian/Ubuntu that would probably be a good starting point for an embedded distribution that cares about space more than providing a complete Python. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dynamic module namspaces
On Sun, Jul 16, 2006 at 11:52:48PM -0700, Josiah Carlson wrote: Andrew Bennetts [EMAIL PROTECTED] wrote: [...] Have you seen the demandload hack that Mercurial uses? You can find it here: http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py You can see an example use of it here: http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py The problem with that particular method is that it requires that you use a string to describe the set of modules you would like to import, rather than a name. I agree, it's ugly. I'd like there to be a nicer solution. In the py3k mailing list I recently described a mechanism (though not implementation) for a somewhat related method to support automatic reloading when a file has changed (for things like web frameworks), but by removing that file change check, you get the equivalent to what you describe, though you don't need to use strings, you can use names... from dynamicreload import DR and used like... DR.os.path.join(...) As long as you use the DR.-prefixed name, you get automatic reloading (if desired) on access. Aside from also being ugly in its own way, a magic prefix like this would add a small performance penalty to places that use it. I believe demandload has the nice feature that once a demandloaded object is accessed, it is replaced with the actual object rather than the demandload gunk, so there's only a one-off performance penalty. The advantage for an interactive command line tool isn't so much memory consumption as speed. Why waste hundreds of milliseconds importing code that isn't used? There's an experimental branch to use the same demandload code in bzr, the reported results are 400ms for 'bzr rocks' down to 100ms, 'bzr root' from 400ms = 200ms, etc. (according to http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) Over half the runtime wasted on importing unused code! There's a definite need for a nice solution to this, and it should be included in the standard batteries that come with Python. Well, just starting up Python without loading any modules can be time consuming; perhaps even dwarfing the few hundred ms saved by dynamic loading. Well, it's only about 10ms on my laptop running Ubuntu (it varies up to 90ms, but I expect that's just noise), and the -S switch makes no obvious difference (tested with python -c ''). 10ms overhead to start python I can live with. It takes about that long run svn --version. If we can address related problems at the same time, like emitting deprecation warnings for accessing certain module attributes, then even better! __deprecated__ = ['name1', ...] Make your dynamic load/reload aware of the __deprecated__ attribute, and you are done. I'm fine with that syntax. But regardless of how it's spelled, I'd really like the standard library or interpreter to have support for it, rather than having a non-standard 3rd-party system for it. I want there to be one-- and preferably only one --obvious way to do it. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dynamic module namspaces
On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote: In an effort to reduce the memory usage used by GTK+ applications written in python I've recently added a feature that allows attributes to be lazy loaded in a module namespace. The gtk python module contains quite a few attributes (around 850) of which many are classes or interfaces (150+) Have you seen the demandload hack that Mercurial uses? You can find it here: http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py You can see an example use of it here: http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py The advantage for an interactive command line tool isn't so much memory consumption as speed. Why waste hundreds of milliseconds importing code that isn't used? There's an experimental branch to use the same demandload code in bzr, the reported results are 400ms for 'bzr rocks' down to 100ms, 'bzr root' from 400ms = 200ms, etc. (according to http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) Over half the runtime wasted on importing unused code! There's a definite need for a nice solution to this, and it should be included in the standard batteries that come with Python. If we can address related problems at the same time, like emitting deprecation warnings for accessing certain module attributes, then even better! -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On Thu, Jun 15, 2006 at 10:33:49PM +0200, Alexander Schremmer wrote: On Thu, 15 Jun 2006 19:00:09 +0200, Jan Claeys wrote: Op di, 13-06-2006 te 10:27 +0200, schreef Alexander Schremmer: Bazaar-NG seems to reach limits already when working on it's own code/repository. Canonical uses bzr to develop launchpad.net, which is a little bit larger dan bzr itself, I suspect...? I don't think so, without having seen the Launchpad code. I assume that Launchpad has less comitters (closed source!) and therefore less change sets and less parallel branches. Actually, Launchpad's got twice as many lines of source (as measured by sloccount), nearly 10 times as many versioned files, and about twice as many revisions as bzr. We typically have 10-20 parallel branches going through our review process at any one time (branches are generally reviewed before they land on the trunk), and probably many others being worked on at any given time. Once I pulled the bzr changesets (1-3 months ago) and it needed 3 hours on a 900 MHz machine with a high-speed ( 50 MBit) internet connection (and it was CPU bound). Note that bzr has gained a lot of speed since then, though. That would have been when it was in weave format? The current knit format doesn't suffer the CPU problems in my experience. It's still very slow over a network because it does a very large number of round trips. There's work to greatly reduce that problem, by pipelining and by reducing the number of HTTP requests (by issuing one request with a range header with many ranges, rather than one request per range!). There are also plans to write a smart server. There's a big focus on performance improvements on the bzr list at the moment, and they seem to be making rapid progress. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] need info for externally maintained modules PEP
On Sun, Apr 09, 2006 at 02:48:47PM -0400, Phillip J. Eby wrote: At 07:56 PM 4/9/2006 +0200, Martin v. Löwis wrote: [...] -1. These aren't external libraries; they are part of Python. They *were* external libraries. Also, many OS vendors nonetheless split the standard library into different system packages, e.g. Debian's longstanding tradition of excising the distutils into a separate python-dev package. Debian has fixed this bug. The entire standard library, including distutils, is now in the python2.4 package. The python2.4-dev package mainly contains header files now. This has been true for some time; it's certainly the case in the current stable release. As much as we might wish that vendors not do these things, they often have practical matters of continuity and documentation to deal with; if they currently have a python-ctypes package, for example, they may wish to maintain that even when ctypes is bundled with 2.5. They can do that just by shipping an empty python-ctypes package that depends on the full python package. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot vs. Windows
Martin v. Löwis wrote: [EMAIL PROTECTED] wrote: [...] So for multiplying this by 8, I would have to create 48 lines of Apache configuration, and use 24 TCP ports. This can be done, but it would take some time to implement. And who is going to look at the 24 pages? This last point is the most important, I think. Most of the time I look at Twisted's buildbot, it's to see at a glance which, if any, builds are broken. I think this is the #1 use case. Second is getting the details of what broke, and who broke it. So massively multiplying the pages seems counter-productive to me. I suspect there's nearly as much advantage to running randomised tests on just one platform as there is on many, so a good trade-off may be to just add one more builder (to each branch) that does -r on just one platform. I'm assuming most of the issues randomisation exposes aren't platform-dependent. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] threadsafe patch for asynchat
Donovan Baarda wrote: On Wed, 2006-02-08 at 02:33 -0500, Steve Holden wrote: Martin v. Löwis wrote: Tim Peters wrote: [...] What is the reason that people want to use threads when they can have poll/select-style message processing? Why does Zope require threads? IOW, why would anybody *want* a threadsafe patch for asynchat? In case the processing of events needed to block? If I'm processing web requests in an async* dispatch loop and a request needs the results of a (probably lengthy) database query in order to generate its output, how do I give the dispatcher control again to process the next asynchronous network event? The usual answer is process the request in a thread. That way the dispatcher can spring to life for each event as quickly as needed. I believe that Twisted does pretty much this with it's deferred stuff. It shoves slow stuff off for processing in a separate thread that re-syncs with the event loop when it's finished. Argh! No. Threading is completely orthogonal to Deferreds. Deferreds are just an abstraction for managing callbacks for an asychronous operation. They don't magically invoke threads, or otherwise turn synchronous code into asynchronous code for you. This seems to be a depressingly common misconception. I wish I knew how to stop it. They're much simpler than people seem to think. They're an object a function returns to say I don't have a result for you yet, but if you attach callbacks to this I'll run those when I do. We've do this because it's much nicer than having to pass callbacks into functions, particularly when you want to deal with chains of callbacks and error handling. There is a single utility function in Twisted called deferToThread that will run a function in a threadpool, and arrange for a Deferred to be fired with the result (in the event loop thread, of course). This is just one of many possible uses for Deferreds, and not an especially common one. I'm happy to provide pointers to several Twisted docs if anyone is at all unclear on this. While they are very useful, I don't think they're an essential part of a minimal Twisted replacement for asyncore/asynchat -- in fact, they'd work just fine with asyncore/asynchat, because they do so little. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str with base
On Tue, Jan 17, 2006 at 09:23:29AM -0500, Jason Orendorff wrote: It seems dumb to support *parsing* integers in weird bases, but not *formatting* them in weird bases. Not a big deal, but if you're going to give me a toy, at least give me the whole toy! The %b idea is a little disappointing in two ways. Even with %b, Python is still dumb by the above criterion. And, I suspect users that don't know about %b are unlikely to find it when they want it. I know I've never looked for it there. I think a method 5664400.to_base(13) sounds nice. It's also a SyntaxError. With the current syntax, it would need to be (5664400).to_base(13) or 5664400 .to_base(13). Why not just make the %b take a base, e.g.: '%13b' % 5664400 wink -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str with base
Guido van Rossum wrote: [...] I'd propose bin() to stay in line with the short abbreviated names. [...] The binary type should have a 0b prefix. It seems odd to me to add both a builtin *and* new syntax for something that's occasionally handy, but only occasionally. If we're going to clutter a module with this function, why not e.g. the math module instead of builtins? I thought the consensus was that we had too many builtins already. Similarly, the need for 0b101 syntax seems pretty low to me when you can already do int(101, 2). -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str with base
On Mon, Jan 16, 2006 at 07:44:44PM -0800, Alex Martelli wrote: Is it finally time in Python 2.5 to allow the obvious use of, say, str(5,2) to give '101', My reaction having read this far was huh?. It took some time (several seconds) before it occurred to me what you wanted str(5,2) to mean, and why it should give '101'. If you'd proposed, say (5).as_binary() == '101', or 5.encode(base2), I wouldn't have been as baffled. Or perhaps even str(5, base=2), but frankly the idea of the string type doing numeric base conversions seems weird to me, rather than symmetric. I wouldn't mind seeing arbitrary base encoding of integers included somewhere, but as a method of str -- let alone the constructor! -- it feels quite wrong. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str with base
On Mon, Jan 16, 2006 at 11:54:05PM -0500, Raymond Hettinger wrote: [...] That suggests that it would be better to simply add an int method: x.convert_to_base(7) This seems clear and simple to me. I like it. I strongly suspect the bright beginners Alex is interested in would have no trouble using it or finding it. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] str with base
On Mon, Jan 16, 2006 at 09:28:10PM -0800, Bob Ippolito wrote:
On Jan 16, 2006, at 9:12 PM, Andrew Bennetts wrote:
[...]
x.convert_to_base(7)
This seems clear and simple to me. I like it. I strongly suspect
the bright
beginners Alex is interested in would have no trouble using it or
finding it.
I don't know about that, all of the methods that int and long
currently have are __special__. They'd really need to start with
Python 2.5 (assuming int/long grow public methods in 2.5) to even
think to look there. A format code or a built-in would be more
likely to be found, since that's how you convert integers to hex and
oct string representations with current Python.
[name for name in dir(0)+dir(0L) if not name.startswith('__')]
[]
I should have said, I'm equally happy with the format code as well (although it
doesn't allow arbitary base conversions, I've never had need for that, so I'm
not too worried about that case). Either option is better than making the str
constructor do relatively rarely used mathematics!
-Andrew.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] building a module catalogue with buildbot
On Thu, Jan 12, 2006 at 07:19:08AM +0100, Martin v. Löwis wrote: Fredrik Lundh wrote: My initial thought was that we could ask alpha testers to run this script on their alpha builds, and report back, but it just struck me that the buildbot already builds stuff on a couple of interesting platforms. Can buildbot deal with custom test/validation scripts, and collect the output somewhere ? Collecting the output is the real challenge. Basically, you are restricted to putting stdout of a process into a file. So a result that is a set of linked HTML files is not supported. A limited solution is just to make the script put the files where they will be published by something other than buildbot. e.g. Twisted's docs are built from SVN by one of our buildslaves, and placed in a directory published at http://twistedmatrix.com/users/warner/doc-latest/ -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot
On Tue, Jan 10, 2006 at 09:15:56AM +0100, Martin v. Löwis wrote: [...] I know I could limit the Twisted webserver to localhost using firewalling/iptables (and I will need to if there is no other option); just having it generate static pages would have been more convenient. For this part at least, it's easier than that; you can pass interface='127.0.0.1' to the relevant listenTCP or TCPServer call, so that it will only bind that port to localhost. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Directory for packages maintained outside the core (was Re: ElementTree - Why not part of the core?)
On Mon, Dec 12, 2005 at 01:32:31PM +, Michael Hoffman wrote: [Hye-Shik Chang] I think contrib is somewhat conventional for the purpose. [Steve Holden] Indeed, but conventionally *all* code in the Python core is contributed, and I think we need a name that differentiates externally-maintained packages from the contributions that are integrated into the core and maintained as part of it. The same could be said of a lot of other projects that use the contrib convention. I have a much better idea of what contrib means than kits or external. I have a much better idea of what contrib means than external, but it's the wrong idea :) contrib implies to me things that are not really a core part of the project (just extras that may perhaps be of use to someone), and so they haven't received the same quality control or integration (e.g. I wouldn't expect to find documentation for it in the standard library reference). Of course, I'm thinking of contrib directories in tar.gz files when I think this, but if I saw a contrib directory in SVN (without having seen this mailing list thread), I'd probably assume the same of it. external is much clearer to me, and has a clear parallel with SVN's svn:external feature. Either way, a simple README.txt in the directory could explain things adequately. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] removing nested tuple function parameters
On Sat, Sep 17, 2005 at 06:20:08PM -0700, Brett Cannon wrote: Is anyone truly attached to nested tuple function parameters; ``def fxn((a,b)): print a,b``? At one of the PyCon sprints Guido seemed okay with just having them removed when Jeremy asked about ditching them thanks to the pain they caused in the AST branch. I personally don't see them being overly useful thanks to assignment unpacking. Plus I don't think they are used very much (gut feeling, though, and not based on any grepping). Would anyone really throw a huge fit if they went away? I am willing to write a PEP for their removal in 2.6 with a deprecation in 2.5 if people are up for it. Otherwise I say they should definitely go in Python 3. Please keep them! Twisted code uses them in places for Deferred callbacks that need to deal with multiple return values. For instance, the twisted.cred.portal.Portal's login method returns Deferred that will be called with a tuple of (interface that the avatar implements, the avatar, a 0-arg logout callable), which naturally leads to functions like: def _cbAuthenticated(self, (iface, avatar, logout)): ... Based on a quick grep, there's well over 80 uses of tuple-unpacking in function declarations in Twisted's code base. There's almost certainly many more outside of Twisted; I know I have some code at work that uses this. So the short answer is: yes, we are attached to it, we will miss it. No guarantees about throwing fits, though wink. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
