Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition
On 12/16/2014 03:09 AM, Barry Warsaw wrote: On Dec 16, 2014, at 02:15 PM, Skip Montanaro wrote: While he doesn't explicitly say so, I got the distinct impression reading his recent blog post that he supports one source, not forked sources. I've ported a fair bit of code, both pure-Python and C extensions, both libraries and applications. For successful library ports to Python 3 that need to remain Python 2 compatible, I would almost always recommend a single source, common dialect, no-2to3 approach. There may be exceptions, but this strategy has proven effective over and over. I generally find I don't need `six` but it does provide some nice conveniences that can be helpful. With something like tox running your test suite, it doesn't even have to be painful to maintain. I'll agree; with tox and some automated CI system like travis or jenkins or whatever, once you've done the port, it's only a minor nuisance to maintain a straddled 2/3 codebase. Programming in only the subset still isn't much fun, but maintenance is slightly easier than I expected it to be. Drive by contributions become slightly harder to accept because they often break 3 compatibility, and contributors are often unable or unwilling to install all the required versions that are tested by tox. - C ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 467: Minor API improvements for bytes bytearray
On 08/17/2014 09:40 PM, Antoine Pitrou wrote: Le 17/08/2014 20:08, Nick Coghlan a écrit : On 18 Aug 2014 09:57, Barry Warsaw ba...@python.org mailto:ba...@python.org wrote: On Aug 18, 2014, at 09:12 AM, Nick Coghlan wrote: I'm talking more generally - do you *really* want to be explaining that bytes behaves like a tuple of integers, while bytes.bytes behaves like a tuple of bytes? I would explain it differently though, using concrete examples. data = bytes(...) for i in data: # iterate over data as integers for i in data.bytes: # iterate over data as bytes But whatever. I just wish there was something better than iterbytes. There's actually another aspect to your idea, independent of the naming: exposing a view rather than just an iterator. So that view would actually be the bytes object done right? Funny :-) Will it have lazy slicing? bytes.sorry()? ;-) - C ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 469: Restoring the iterkeys/values/items() methods
On 04/19/2014 07:41 AM, Kristján Valur Jónsson wrote: Wouldn't iterkeys simply be an alias for keys and so on? I'm +1 on that. It is a signigificant portion of the incompatibility, and seems like such a minor concession to compatibility to make. K FWIW, I'm +1 on this and other minor changes and concessions to bw compat like this improve the subset language that 2/3 straddling projects need to use. - C ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 15:44 +1000, Nick Coghlan wrote: On 16 July 2013 13:16, Chris McDonough chr...@plope.com wrote: I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have PEP8 modes turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as import pip rather than import _pip and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up. Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013. However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate). Well, I know it's arguing with the umpire at this point, but PEP8 prescriptionism (great word by the way!) is de rigeur these days. It's a python-dev perogative to ignore that, but it has consequences that reverberate further than this maillist. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the non-internal API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to PEP8-ify my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. - C On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote: Private interfaces Internal or Non-public Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. How about: All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute. While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring). As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote: If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. I have better things to do than PEP8-ify old, working, stable code is a perfectly acceptable answer. I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so might be an even better one :-) Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code. /scarcasm - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 12:34 +1000, Nick Coghlan wrote: How do get from If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring to updating 500k lines of public code? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. I may be wrong, but as I see it this is a decision to help (although only conditionally!) a theoretical IDE user or an agressive user of dir() at the expense of effectively either 1) making existing maintainers revisit existing code, or 2 ) making maintainers continually explain to users that their codebase is not PEP8 compliant. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. Fair enough, technically. But historical reality is that a lot of people just haven't read much past the title: Style Guide for Python Code. Or at least they appear to have no concept that it's about code *in the stdlib only*. People take it way, way too seriously for non-stdlib code. But they do, and it takes time to manage that. Given that misunderstanding, is there a way to divorce stdlib-centric guidelines like the one being discussed now from PEP8-ness? I don't think any amount of marketing effort or reasoned explanation is going to separate PEP8 from correct code in people's minds at this point. - C ps.. the real irritant behind my minirant is this: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 13:11 +1000, Nick Coghlan wrote: On 16 July 2013 13:02, Chris McDonough chr...@plope.com wrote: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as all Python code in the world should follow these rules cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have PEP8 modes turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 443 - Single-dispatch generic functions (including ABC support)
On Fri, 2013-05-31 at 03:05 +0200, Łukasz Langa wrote: On 31 maj 2013, at 01:51, Łukasz Langa luk...@langa.pl wrote: Back to the point, though. I don't feel we should complicate the code, tests and documentation by introducing special handling for methods. In terms of pure type-driven single dispatch, we have a solution that was intentionally simple from the get-go. The next step will be predicate dispatch anyway ;)) What do you think? +1. It's incredibly useful and easy to document as-is. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Fri, 2013-05-24 at 15:56 -0400, Barry Warsaw wrote: Here's something that seems to come up from time to time in Debian. Take a Python application like tox, nose, or pyflakes. Their executables work with both Python 2 and 3, but require a #! line to choose which interpreter to invoke. You probably already know this, but I'll mention it anyway. This probably matters a lot for nose and pyflakes, but I'd say that for tox it should not, it basically just scripts execution of shell commands. I'd think maybe in cases like tox (and others that are compatible with both Python 2 and 3) the hashbang should just be set to #!/usr/bin/python unconditionally. Maybe we could also think about modifying pyflakes so that it can validate both 2 and 3 code (choosing one or the other based on a header line in the validated files and defaulting to the version of Python being run). This is kind of the right thing anyway. Nose is a bit of a special case. I personally never run nosetests directly, I always use setup.py nosetests, which makes it not matter. In general, I'd like to think that scripts that get installed to global bindirs will execute utilities that are useful independent of the version of Python being used to execute them. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Sat, 2013-05-25 at 17:57 +1000, Nick Coghlan wrote: I think the simplest thing to do is just append the 3 to the binary name (as we do ourselves for pydoc) and then abide by the recommendations in PEP 394 to reference the correct system executable. I'm curious if folks have other concrete examples of global bindir executables other than nosetests and pydoc that need to be disambiguated by Python version. I'd hate to see it become standard practice to append 3 to scripts generated by packages which happen to use Python 3, as it will just sort of perpetuate its otherness. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.7.4
On Sat, 2013-01-19 at 14:30 -0500, Benjamin Peterson wrote: It's been almost a year since 2.7.3, so it's time for another 2.7 bugfix release. 2013-02-02 - 2.7.4 release branch created; rc released 2013-02-16 - 2.7.4 released Does this work for you, Martin and Ned? I have a pet issue that has a patch that requires application to the 2.7 branch, if anyone would be kind enough to do it: http://bugs.python.org/issue15881 It has already been applied to various 3.X branches. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 423 : naming conventions and recipes related to packaging
On 06/28/2012 06:36 AM, Benoît Bryon wrote: Le 27/06/2012 13:34, Antoine Pitrou a écrit : Similarly, I think the section about private projects (Private (including closed-source) projects use a namespace) should be removed. It is not our duty to promote naming standards for private (i.e. internal) projects. The intention in the proposed PEP is to promote standards for general Python usage, which implicitely includes both public and private use. The proposed PEP tries to explain how the conventions apply in most use cases. Public and private scopes are mentioned explicitely because they were identified as valuable use cases. Here are some reasons why the private code use case has been identified as valuable: * New Python developers (or more accurately new distribution authors) may wonder What makes a good name?, even if they are working in a private area. Guidelines covering private code would be welcome. * At work, I already had discussions about naming patterns for closed source projects. These discussions consumed some energy made the team focus on some less valuable topics. We searched for an official convention and didn't find one. We made choices but none of us was really satisfied about it. An external arbitration from a trusted authority would have been welcome, even if we were making closed-source software. * As Paul said, personal code often ends up more widely used than originally envisaged. So following the convention from the start may help. * Here, the PEP already covers (or tries to) most public code use cases. It's quite easy to extend it to private code. I feel drawbacks are negligible compared to potential benefits. .. note:: IMHO, main drawback is read this long document. * Isn't it obvious that, at last, people do what they want to in private code? In fact, they also do in public code. I mean the document is an informational PEP. It recommends to apply conventions but the actual choice is left to developers. That said, would the changes below improve the document? * Keep the parts about private and closed-source code, but add a note to insist on in private code, you obviously do what you want to and be aware that personal code often ends up more widely used than originally envisaged. * At the beginning of the document, add a section like http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds Another option would have been to deal with general Python code and don't mention public and private areas, i.e. implicitely cover both. I haven't followed this way because it is implicit. Also, I don't see what's so important about using your company's name as a top-level namespace. You don't need it for conflict avoidance: you can just as well use distinctive project names. Using company's name as top-level namespace has been proven a good practice: * avoids clashes and confusion with public projects, i.e. don't create a cms private project because there could be a cms project on PyPI. * makes it easy to release the code as open-source: don't change the project name. * if there is no reason at all for the project to contain the company name (i.e. the project is not specific to the company), why not realeasing it as open source? (with a one-level name) Using company's name is not the only option. But, as far as I know, it fits most use cases, which is enough (and required) for a convention. Another option is to use any arbitrary name as top-level namespace. You can. If an arbitrary name seems obvious to you, feel free to use it. But, in most cases, company's name is an obvious choice. So, would you appreciate a change so that: * company name is recommended as a best practice. * but any arbitrary name can be used. Could be something in: 1. For private projects, use company name (or any unique arbitrary name) as top-level namespace. 2. For private projects, use any arbitrary name (company name is generally a good choice) as top-level namespace. 3. For private projects, use a top-level namespace (company name is generally a good choice, but you can use any unique arbitrary name). It's probably always a reasonable idea to use a brand-prefixed namespace for *private* packages but in my experience it's almost always a bad idea to publish any set of otherwise-unrelated packages that share a branded namespace prefix to PyPI. I know this because I've been involved with it at least twice with zope. and repoze. brands/namespaces. The meaning of both of those namespaces has become soft over time and both now mean basically this code was created by a group of people instead of this code is useful under a circumstance or for a purpose. Those namespaces are both the moral equivalent of a garbage barge class in development: code related to the namespace might do anything (zope now means a company and two completely different application
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 04:45 AM, Nick Coghlan wrote: On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonoughchr...@plope.com wrote: All of these are really pretty minor issues compared with the main benefit of not needing to ship everything with everything else. The killer feature is that developers can specify dependencies and users can have those dependencies installed automatically in a cross-platform way. Everything else is complete noise if this use case is not served. Cool. This is the kind of thing we need recorded in a PEP - there's a lot of domain knowledge floating around in the heads of packaging folks that needs to be captured so we can know *what the addition of packaging to the standard library is intended to fix*. And, like it or not, setuptools has a serious PR problem due to the fact it monkeypatches the standard library, uses *.pth files to alter sys.path for every installed application by default, actually *uses* the ability to run code in *.pth files and has hard to follow documentation to boot. I *don't* trust that I fully understand the import system on any machine with setuptools installed, because it is demonstrably happy to install state to the file system that will affect *all* Python programs running on the machine. I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff on the default sys.path of the system Python of the box that's related to their software's concerns only. I don't understand why people accept this but get crazy about the fact that installing a setuptools distribution using easy_install changes the default sys.path. Installing a distribution will change behavior whether or not sys.path is changed as a result. That's its purpose. The code that runs in the .pth *file* (there's only one that matters: easy_install.pth) just mutates sys.path. The end result is this: if you understand how sys.path works, you understand how eggs work. Each egg is addded to sys.path. That's all there is to it. It's the same as manually mutating a global PYTHONPATH, except you don't need to do it. And note that this is not setuptools in general. It's easy_install in particular. Everything you've brought up so far I think is limited to easy_install. It doesn't happen when you use pip. I think it's a mistake that pip doesn't do it, but I think you have to make more accurate distinctions. A packaging PEP needs to explain: - what needs to be done to eliminate any need for monkeypatching - what's involved in making sure that *.pth are *not* needed by default - making sure that executable code in implicitly loaded *.pth files isn't used *at all* I'll note that these goals are completely sideways to any actual functional goal. It'd be a shame to have monkeypatching going on, but the other stuff I don't think are reasonable goals. Instead they represent fears, and those fears just need to be managed. I *think* trying to achieve this is actually the genesis of the original distribute fork, that subsequently became distutils2 as Tarek discovered how much of the complexity in setuptools was actually due to the desire to *not* officially fork distutils (and instead monkeypatch it, effectively creating a runtime fork). However, for those of us that weren't directly involved, this is all still a strange mystery dealt with by other people. I've cribbed together bits and pieces just from following the fragments of the discussions that have happened on python-dev and at PyCon US, but if we want the madness to ever stop, then *the problems with the status quo* need to be written down so that other core developers can understand them. It'd also be useful if other core developers actually tried to use setuptools in anger. That'd be a good start towards understanding some of its tradeoffs. People can write this stuff down til they're blue in the face, but if core devs don't try the stuff, they'll always fear it. In fact, I just remembered that Tarek *has* written a lot of this down, just not in PEP form: http://www.aosabook.org/en/packaging.html Cool. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 08:21 AM, Nick Coghlan wrote: Installing a distribution will change behavior whether or not sys.path is changed as a result. That's its purpose. No it won't. An ordinary package will only change the behaviour of Python applications that import a package by that name. Other Python applications will be completely unaffected (as it should be). If a Python application is effected by a change to sys.path which doesn't impact modules it uses, then that Python application is plain broken, because the developer of that application cannot make assumptions about what a user does to sys.path unrelated to the modules it requires. This is completely independent of easy_install. Any Python application is going to be effected by the installation of a distribution that does impact modules it imports, whether sys.path is used to change the working set of modules or not. So what concrete situation are we actually talking about here? The code that runs in the .pth *file* (there's only one that matters: easy_install.pth) just mutates sys.path. The end result is this: if you understand how sys.path works, you understand how eggs work. Each egg is addded to sys.path. That's all there is to it. It's the same as manually mutating a global PYTHONPATH, except you don't need to do it. Yes, it's the same as mutating PYTHONPATH. That's a similarly bad system global change. Individual libraries do not have the right to change the sys.path seen on initialisation by every other Python application on that system. Is it reasonable to even assume there is only one-sys.path-to-rule-them-all? And that users install the set of libraries they need into a common place? This quickly turns into failure, because Python is used for many, many tasks, and those tasks sometimes *require conflicting versions of libraries*. This is the root cause of why virtualenv exists and is popular. The reason it's disappointing to see OS vendors mutating the default sys.path is because they put *very old versions of very common non-stdlib packages* (e.g. zope.interface, lxml) on sys.path by default. The path is tainted out of the box for anyone who wants to use the system Python for development of newer software. So at some point they invariably punt to virtualenv or a virtualenv-like system where the OS-vendor-provided path is not present. If Python supported the installation of multiple versions of the same module and versioned imports, both PYTHONPATH and virtualenv would be much less important. But given lack of enthusiasm for that, I don't think it's reasonable to assume there is only one sys.path on every system. I sympathize, however, with Oscar's report that PYTHONPATH can't the setuptools-derived path. That's indeed a mistake that a future tool should not make. And note that this is not setuptools in general. It's easy_install in particular. Everything you've brought up so far I think is limited to easy_install. It doesn't happen when you use pip. I think it's a mistake that pip doesn't do it, but I think you have to make more accurate distinctions. What part of PR problem was unclear? setuptools and easy_install are inextricably linked in everyone's minds, just like pip and distribute. Hopefully for the purposes of the discussion, folks here can make the mental separation between setuptools and easy_install. We can't help what other folks think in the meantime, certainly not solely by making technological compromises anyway. A packaging PEP needs to explain: - what needs to be done to eliminate any need for monkeypatching - what's involved in making sure that *.pth are *not* needed by default - making sure that executable code in implicitly loaded *.pth files isn't used *at all* I'll note that these goals are completely sideways to any actual functional goal. It'd be a shame to have monkeypatching going on, but the other stuff I don't think are reasonable goals. Instead they represent fears, and those fears just need to be managed. No, they reflect the mindset of someone with configuration management and auditing responsibilities for shared systems with multiple applications installed which may be written in a variety of languages, not just Python. You may not care about those people, but I do. I care about deploying Python-based applications to many platforms. You care about deploying multilanguage-based applications to a single platform. There's going to be conflict there. My only comment on that is this: Since this is a problem related to the installation of Python distributions, it should deal with the problems that Python developers have more forcefully than non-Python developers and non-programmers. It'd also be useful if other core developers actually tried to use setuptools in anger. That'd be a good start towards understanding some of its tradeoffs. People can write this stuff down til they're blue in the face, but if core devs don't
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 09:29 AM, Nick Coghlan wrote: My only comment on that is this: Since this is a problem related to the installation of Python distributions, it should deal with the problems that Python developers have more forcefully than non-Python developers and non-programmers. Thanks to venv, there's an alternative available that may be able to keep both of us happy: split the defaults. For system installs, adopt a vendor-centric, multi-language, easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding *.pth files by unpacking eggs to the file system). For venv installs, do whatever is most convenient for pure Python developers (e.g. leaving eggs packed and using *.pth files to extend sys.path within the venv). I'd like to agree with this, but I think there's a distinction that needs to be made here that's maybe not obvious to everyone. A tool to generate an OS-specific system package from a Python library project should be unrelated to a Python distribution *installer*. Instead, you'd use related tools that understood how to unpack the distribution packaging format to build one or more package structures. The resulting structures will be processed and then eventually installed by native OS install tools. But the Python distribution installer (e.g easy_install, pip, or some future similar tool) would just never come into play to create those structures. The Python distribution installer and the OS-specific build tool might share code to introspect and unpack files from the packaging format, but they'd otherwise have nothing to do with one another. This seems like the most reasonable separation of concerns to me anyway, and I'd be willing to work on the code that would be shared by both the Python-level installer and by OS-level packaging tools. One of Python's great virtues is its role as a glue language, and part of being an effective glue language is playing well with others. That should apply to packaging distribution as well, not just to runtime bindings to tools written in other languages. When we add the scientific users into the mix, we're actually getting to a *third* audience: multi-language developers that want to use *Python's* packaging utilities for their source and binary distribution formats. The Python community covers a broad spectrum of use cases, and I suspect that's one of the big reasons packaging can get so contentious - the goals end up being in direct conflict. Currently, I've identified at least half a dozen significant communities with very different needs (the names aren't meant to be all encompassing, just good representatives of each category, and many individuals will span multiple categories depending on which hat they're wearing at the time): Library authors: just want to quickly and easily publish their work on the Python package index in a way that is discoverable by others and allows feedback to reach them at their development site Web developers: creators of Python applications, relying primarily on other Python software and underlying OS provided functionality, potentially with some native extensions, that may need to run on multiple platforms, but can require installation using a language specific mechanism by technical staff Rich client developers: creators of Python applications relying primarily on other Python software and underlying OS provided functionality, potentially with native extensions, that need to run on multiple platforms, but must be installed using standard system utilities for the benefit of non-technical end users Enterprise developers: creators of Python or mixed language applications that need to integrate with corporate system administration policies (including packaging, auditing and configuration management) Scientists: creators of Python data analysis and modelling applications, with complex dependencies on software written in a variety of other languages and using various build systems Python embedders: developers that embed a Python runtime inside a larger application I think we'll also need to put some limits on the goal independent of the union of everything all the audiences require. Here's some scope suggestions that I believe could be shared by all of the audiences you list above except for embedders; I think that use case is pretty much separate. It might also leave rich client developers wanting, but no more than they're already wanting. - Install code that can *later be imported*. This could be pure Python code or C code which requires compilation. But it's not for the purpose of compiling and installing completely arbitrary C code to arbitrary locations, it's ust written for the purpose of compiling C code which then *lives in the installed distribution* to provide an importable Python module that lives in the same distribution with logic. - Install console scripts which are shell-scripts/batch-files that cause some logic written in Python to get run as
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 10:30 AM, Nick Coghlan wrote: A tool to generate an OS-specific system package from a Python library project should be unrelated to a Python distribution *installer*. Instead, you'd use related tools that understood how to unpack the distribution packaging format to build one or more package structures. The resulting structures will be processed and then eventually installed by native OS install tools. But the Python distribution installer (e.g easy_install, pip, or some future similar tool) would just never come into play to create those structures. The Python distribution installer and the OS-specific build tool might share code to introspect and unpack files from the packaging format, but they'd otherwise have nothing to do with one another. This seems like the most reasonable separation of concerns to me anyway, and I'd be willing to work on the code that would be shared by both the Python-level installer and by OS-level packaging tools. Right, but if the standard library grows a dist installer (and I think it eventually should), we're going to need to define how it should behave when executed with the *system* Python. That will give at least 3 mechanisms for Python code to get onto a system: 1. Python dist - converter - system package - system Python path 2. Python dist - system Python installer - system Python path 3. Python dist - venv Python installer - venv Python path While I agree that path 2 should be discouraged for production systems, I don't think it should be prevented altogether (since it can be very convenient on personal systems). I'm not sure under what circumstance 2 and 3 wouldn't do the same thing. Do you have a concrete idea? As far as the scope of the packaging utilities and what they can install goes, I think the distutils2 folks have done a pretty good job of defining that with their static metadata format: http://alexis.notmyidea.org/distutils2/setupcfg.html#files Yeah definitely a good start. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 11:37 AM, PJ Eby wrote: On Jun 21, 2012 11:02 AM, Zooko Wilcox-Oapos;Hearn zo...@zooko.com mailto:zo...@zooko.com wrote: Philip J. Eby provisionally approved of one of the patches, except for some specific requirement that I didn't really understand how to fix and that now I don't exactly remember: http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html I don't remember either; I just reviewed the patch and discussion, and I'm not finding what the holdup was, exactly. Looking at it now, it looks to me like a good idea... oh wait, *now* I remember the problem, or at least, what needs reviewing. Basically, the challenge is that it doesn't allow an .egg in a PYTHONPATH directory to take precedence over that *specific* PYTHONPATH directory. With the perspective of hindsight, this was purely a transitional concern, since it only *really* mattered for site-packages; anyplace else you could just delete the legacy package if it was a problem. (And your patch works fine for that case.) However, for setuptools as it was when you proposed this, it was a potential backwards-compatibility problem. My best guess is that I was considering the approach for 0.7... which never got any serious development time. (It may be too late to fix the issue, in more than one sense. Even if the problem ceased to be a problem today, nobody's going to re-evaluate their position on setuptools, especially if their position wasn't even based on a personal experience with the issue.) A minor backwards incompat here to fix that issue would be appropriate, if only to be able to say hey, that issue no longer exists to folks who condemn the entire ecosystem based on that bug. At least, that is, if there will be another release of setuptools. Is that likely? - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 11:45 AM, PJ Eby wrote: On Jun 21, 2012 10:12 AM, Chris McDonough chr...@plope.com mailto:chr...@plope.com wrote: - Install package resources, which are non-Python source files that happen to live in package directories. I love this phrasing, by the way (non-Python source files). A pet peeve of mine is the insistence by some people that such files are data and don't belong in package directories, despite the fact that if you gave them a .py extension and added data=... around them, they'd be considered part of the code. A file's name and internal format aren't what distinguishes code from data; it's the way it's *used* that matters. I think packaging has swung the wrong way on this particular point, and that resources and data files should be distinguished in setup.cfg, with sysadmins *not* being given the option to muck about with resources -- especially not to install them in locations where they might be mistaken for something editable. +1. A good number of the package resource files we deploy are not data files at all. In particular, a lot of them are files which represent HTML templates. These templates are exclusively the domain of the software being installed, and considering them explicitly more editable than the Python source they sit next to in the package structure is a grave mistake. They have exactly the same editability candidacy as the Python source files they are mixed in with. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 12:26 PM, PJ Eby wrote: On Thu, Jun 21, 2012 at 11:50 AM, Chris McDonough chr...@plope.com mailto:chr...@plope.com wrote: On 06/21/2012 11:37 AM, PJ Eby wrote: On Jun 21, 2012 11:02 AM, Zooko Wilcox-Oapos;Hearn zo...@zooko.com mailto:zo...@zooko.com mailto:zo...@zooko.com mailto:zo...@zooko.com wrote: Philip J. Eby provisionally approved of one of the patches, except for some specific requirement that I didn't really understand how to fix and that now I don't exactly remember: http://mail.python.org/__pipermail/distutils-sig/2009-__January/010880.html http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html I don't remember either; I just reviewed the patch and discussion, and I'm not finding what the holdup was, exactly. Looking at it now, it looks to me like a good idea... oh wait, *now* I remember the problem, or at least, what needs reviewing. Basically, the challenge is that it doesn't allow an .egg in a PYTHONPATH directory to take precedence over that *specific* PYTHONPATH directory. With the perspective of hindsight, this was purely a transitional concern, since it only *really* mattered for site-packages; anyplace else you could just delete the legacy package if it was a problem. (And your patch works fine for that case.) However, for setuptools as it was when you proposed this, it was a potential backwards-compatibility problem. My best guess is that I was considering the approach for 0.7... which never got any serious development time. (It may be too late to fix the issue, in more than one sense. Even if the problem ceased to be a problem today, nobody's going to re-evaluate their position on setuptools, especially if their position wasn't even based on a personal experience with the issue.) A minor backwards incompat here to fix that issue would be appropriate, if only to be able to say hey, that issue no longer exists to folks who condemn the entire ecosystem based on that bug. At least, that is, if there will be another release of setuptools. Is that likely? Yes. At the very least, there will be updated development snapshots (which are what buildout uses anyway). (Official releases are in a bit of a weird holding pattern. distribute's versioning scheme leads to potential confusion: if I release e.g. 0.6.1, then it sounds like it's a lesser version than whatever distribute is up to now. OTOH, releasing a later version number than distribute implies that I'm supporting their feature enhancements, and I really don't want to add new features to 0.6... but don't have time right now to clean up all the stuff I started in the 0.7 line either, since I've been *hoping* that the work on packaging would make 0.7 unnecessary. And let's not even get started on the part where system-installed copies of distribute can prevent people from downloading or installing setuptools in the first place.) Welp, I don't want to get in the middle of that whole mess. But maybe the distribute folks would be kind enough to do a major version bump in their next release; e.g. 1.67 instead of 0.67. That said, I don't think anyone would be confused by overlapping version numbers between the two projects. It's known that they have been diverging for a while. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/21/2012 01:20 PM, Tarek Ziadé wrote: On 6/21/12 6:44 PM, Chris McDonough wrote: Yes. At the very least, there will be updated development snapshots (which are what buildout uses anyway). (Official releases are in a bit of a weird holding pattern. distribute's versioning scheme leads to potential confusion: if I release e.g. 0.6.1, then it sounds like it's a lesser version than whatever distribute is up to now. OTOH, releasing a later version number than distribute implies that I'm supporting their feature enhancements, and I really don't want to add new features to 0.6... but don't have time right now to clean up all the stuff I started in the 0.7 line either, since I've been *hoping* that the work on packaging would make 0.7 unnecessary. And let's not even get started on the part where system-installed copies of distribute can prevent people from downloading or installing setuptools in the first place.) Welp, I don't want to get in the middle of that whole mess. But maybe the distribute folks would be kind enough to do a major version bump in their next release; e.g. 1.67 instead of 0.67. That said, I don't think anyone would be confused by overlapping version numbers between the two projects. Oh yeah no problem, if Philip backports all the things we've done like Py3 compat, and bless more people to maintain setuptools, we can even discontinue distribute ! If not, I think you are just being joking here -- we don't want to go back into the lock situation we've suffered for many years were PJE is the only maintainer then suddenly disappears for a year, telling us no one that is willing to maintain setuptools is able to do so. (according to him) It's known that they have been diverging for a while. Yeah the biggest difference is Py3 compat, other than that afaik I don't think any API has been removed or modified. In my opinion, distribute is the only project that should go forward since it's actively maintained and does not suffer from the bus factor. I'm not too interested in the drama/history of the fork situation so I don't care whether setuptools has the fix or distribute has it or both have it, but being able to point at some package which doesn't prevent folks from overriding sys.path ordering using PYTHONPATH would be a good thing. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of packaging in 3.3
On 06/20/2012 11:57 PM, Nick Coghlan wrote: On Thu, Jun 21, 2012 at 3:29 AM, PJ Ebyp...@telecommunity.com wrote: On Wed, Jun 20, 2012 at 9:02 AM, Nick Coghlanncogh...@gmail.com wrote: On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrousolip...@pitrou.net wrote: Agreed, especially if the proven in the wild criterion is required (people won't rush to another third-party distutils replacement, IMHO). The existence of setuptools means that proven in the wild is never going to fly - a whole lot of people use setuptools and easy_install happily, because they just don't care about the downsides it has in terms of loss of control of a system configuration. Um, this may be a smidge off topic, but what loss of control are we talking about here? AFAIK, there isn't anything it does that you can't override with command line options or the config file. (In most cases, standard distutils options or config files.) Do you just mean that most people use the defaults and don't care about there being other options? And if that's the case, which other options are you referring to? No, I mean there are design choices in setuptools that explain why many people don't like it and are irritated when software they want to use depends on it without a good reason. Clearly articulating the reasons that just include setuptools is no longer being considered as an option should be one of the goals of any PEPs associated with adding packaging back for 3.4. The reasons I'm personally aware of: - it's a unilateral runtime fork of the standard library that bears a lot of responsibility for the ongoing feature freeze in distutils. Standard assumptions about the behaviour of site and distutils cease to be valid once setuptools is installed - overuse of *.pth files and the associated sys.path changes for all Python programs running on a system. setuptools gleefully encourages the inclusion of non-trivial code snippets in *.pth files that will be executed by all programs. - advocacy for the egg format and the associated sys.path changes that result for all Python programs running on a system - too much magic that is enabled by default and is hard to switch off (e.g. http://rhodesmill.org/brandon/2009/eby-magic/) All of these are really pretty minor issues compared with the main benefit of not needing to ship everything with everything else. The killer feature is that developers can specify dependencies and users can have those dependencies installed automatically in a cross-platform way. Everything else is complete noise if this use case is not served. IMO, the second and third things you mention above (use of pth files and eggs) are actually features when compared against the result of something like pip, which installs things using --single-version-externally-managed and then tries to manage the resulting potentially-intertwined directories. Eggs are *easier* to manage than files potentially overlapping files and directories installed into some other directory. Either they exist or they don't. Either they're mentioned in a .pth file or they aren't. It's not really that hard. In any case, any tool that tries to manage distribution installation will need somewhere to keep distribution metadata. It's a minor mystery to me why people think it could be done much better than in something very close to egg format. System administrators (and developers that think like system administrators when it comes to configuration management) *hate* what setuptools (and setuptools based installers) can do to their systems. It doesn't matter that package developers don't *have* to do those things - what matters is that the needs and concerns of system administrators simply don't appear to have been anywhere on the radar when setuptools was being designed. (If those concerns actually were taken into account at some point, it's sure hard to tell from the end result and the choices of default behaviour) I think you mean easy_install here. And I guess you mean managing .pth files. Note that if you use pip, neither thing needs to happen. And even easy_install lets you install a distribution that way (with --single-version-externally-managed). So I think, as you mention, this is a matter of defaults (tool and or flag defaults) rather than core functionality. setuptools is a masterful achievement built on shaky foundations that will work most of the time. However, when it doesn't work, you're probably screwed, and as soon as it's present on a system, you know that your assumptions about understanding the Python interpreter's startup sequences are probably off. It's true setuptools is based on shaky foundations. The rest of the stuff you say above is pretty darn specious, I think. The efforts around distutils2/packaging have been focused on taking the time to *fix the foundations first* rather than accepting the inevitable shortcomings of trying to build something in the middle of a swamp. If the
Re: [Python-Dev] Status of packaging in 3.3
On 06/19/2012 05:46 PM, Éric Araujo wrote: Hi all, We need to make a decision about the packaging module in Python 3.3. Please read this message and breathe deeply before replying :) ... With beta coming, a way to deal with that unfortunate situation needs to be found. We could (a) grant an exception to packaging to allow changes after beta1; (b) keep packaging as it is now under a provisional status, with due warnings that many things are expected to change; (c) remove the unstable parts and deliver a subset that works (proposed by Tarek to the Pyramid author on distutils-sig); (d) not release packaging as part of Python 3.3 (I think that was also suggested on distutils-sig last month). I think it'd be very wise to choose (d) here. We've lived so long without a credible packaging story that waiting one (or even two) more major release cycles isn't going to make a huge difference in the long run but including a version of packaging now which gets fixed in a rush would probably end up muddying the already dark waters of Python software distribution. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 updated
On Sun, 2012-03-04 at 17:34 +1000, Nick Coghlan wrote: My rewritten version of PEP 414 is now up (http://www.python.org/dev/peps/pep-0414/). It describes in detail a lot more of the historical background that was taken as read when Guido accepted the PEP. Just as support for string exceptions was eliminated from Python 2 using the normal deprecation process, support for redundant string prefix characters (specifically, B, R, u, U) may be eventually eliminated from Python 3, regardless of the current acceptance of this PEP. We might need to clarify the feature's longevity. I take the above to mean that use of u'' and/or U'' won't emit a deprecation warning in 3.3. But that doesn't necessarily mean its usage won't emit a deprecation warning in 3.4 or 3.5 or 3.6, or whenever it feels right? Does that sound about right? - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Fri, 2012-03-02 at 14:41 -0500, Barry Warsaw wrote: On Mar 02, 2012, at 02:48 PM, Nick Coghlan wrote: Consider: an application that uses 8-bit strings everywhere and blows up on non-ASCII data in Python 2 has at least a fighting chance to run unmodified *and* handle Unicode properly on Python 3. Because unicode literals are gone, a Unicode-aware Python 2 application currently has *no* chance to run unmodified on Python 3. On its face, this statement is incorrect. It *might* be accurate if qualified by saying a Unicode-aware Python 2 *web* application. I say might because I'm not an expert on web frameworks so I defer to those who are. It certainly can't be applied to the entire universe of Unicode-aware Python 2 applications. FWIW, I think this issue's webness may be overestimated. There happens to be lots and lots of existing UI code which contains complex interactions between unicode literals and nonliterals in web apps, but there's also likely lots of nonweb code that has the same issue. If e.g. wxPython had already been ported, I think you'd be hearing the same sorts of things from folks that had investments in existing Python-2-compatible code when trying to port stuff to Py3 (at least if they wanted to run on both Python 2 and Python 3 within the same codebase). - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Fri, 2012-03-02 at 15:39 -0500, Barry Warsaw wrote: On Mar 02, 2012, at 03:13 PM, Chris McDonough wrote: FWIW, I think this issue's webness may be overestimated. There happens to be lots and lots of existing UI code which contains complex interactions between unicode literals and nonliterals in web apps, but there's also likely lots of nonweb code that has the same issue. If e.g. wxPython had already been ported, I think you'd be hearing the same sorts of things from folks that had investments in existing Python-2-compatible code when trying to port stuff to Py3 (at least if they wanted to run on both Python 2 and Python 3 within the same codebase). Okay, I just want to be very careful about the message we're sending here, because I think many libraries and applications will work fine with the facilities available in today's stable releases, i.e. unicode_literals and b-prefixes. For these, there's no need to define native strings, nor do they require language constructs above what's already available. Although the change makes it possible, and it is very useful for very low level WSGI apps, the issue this change addresses really isn't really 100% about needing to define native strings. It's also just preservation of a resource in pretty short supply: developer energy. You will probably need to modify less code when taking piece of software that currently runs on Python 2 and changing it so that it runs on both Python 2 and Python 3, without needing to worry over the unintended consequences of using a unicode_literals future import or replacing existing u'' with a function call. This, IMO, can only be a good thing, because the nominal impact of some future user who must now understand u'' syntax is (again IMO) not as consequential as that user having less software to choose from because porting to Python 3 was just that much harder for existing Python 2 developers. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
On Tue, 2012-02-28 at 15:54 -0500, Benjamin Peterson wrote: 2012/2/28 Ethan Furman et...@stoneleaf.us: Here's what I know: We don't add features to bug-fix releases. u'' is considered a feature. By not backporting to 3.1 and 3.2 we are not easing the migration pains from 2.x. Here's what I don't know: Why is readding u'' a feature and not a bug? (Just had a thought about this -- because the removal of u'' is documented.) Because it's a new thing which doesn't fix obviously broken behavior. If there is already a FAQ entry feel free to point me to it, but I would still be curious why, in this instance, practicality does not beat purity? Because it's practical not to break bugfix releases with new features. This change, by its nature, cannot break old programs. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting PEP 414
On Tue, 2012-02-28 at 16:48 -0500, Barry Warsaw wrote: On Feb 28, 2012, at 03:54 PM, Benjamin Peterson wrote: If there is already a FAQ entry feel free to point me to it, but I would still be curious why, in this instance, practicality does not beat purity? Because it's practical not to break bugfix releases with new features. And because now your code is incompatible with three micro-release versions (3.2.0, 3.2.1, and 3.2.2), two of which are bug fix releases. Which means for example, you can't be sure which version of which distro your code will work on. That I do sympathize with. Doesn't anybody else remember the True/False debacle in 2.2.1? I do. It was slightly different than this because the feature was added twice, once in 2.2.1 with direct aliases to 0 and 1, which was found to be lacking, and then later again in 2.3 with explicit types, so it was sort of an extended-timeframe unpleasantness, and the feature's minor-dot-introduction was only a contributing factor, IIRC. But yeah. A year from now I wouldn't remember which version of 3.2 got a new feature, and neither would anybody else. The no-new-features guidelines are useful in the real world this way, because they represent a coherent policy, as tempting as it is to just jam it in. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 12:41 -0500, R. David Murray wrote:
On Mon, 27 Feb 2012 09:05:54 -0800, Ethan Furman et...@stoneleaf.us wrote:
Martin v. Löwis wrote:
Am 26.02.2012 07:06, schrieb Nick Coghlan:
On Sun, Feb 26, 2012 at 1:13 PM, Guido van Rossum gu...@python.org
wrote:
A small quibble: I'd like to see a benchmark of a 'u' function
implemented in C.
Even if it was quite fast, I don't think such a function would bring
the same benefits as restoring support for u'' literals.
You claim that, but your argument doesn't actually support that claim
(or I fail to see the argument).
Python 2.6 code:
this = u'that'
Python 3.3 code:
this = u('that')
Not source compatible, not elegant. (Even though 2to3 could make this
fix, it's still kinda ugly.)
Eh? The 2.6 version would also be u('that'). That's the whole point
of the idiom. You'll need a better counter argument than that.
The best argument is that there already exists tons and tons of Python 2
code that already does:
u'that'
Needing to change it to:
u('that')
1) Requires effort on the part of a from-Python-2-porter to service
the aesthetic and populist goal of not having an explicit
but redundant-under-Py3 literal syntax that says this is text.
2) Won't atually meet the aesthetic goal, as
it's uglier and slower under *both* Python 2 and Python 3.
So the populist argument remains.. it's too confusing for people who
learn Python 3 as a new language to have a redundant syntax. But we've
had such a syntax in Python 2 for years with b'', and, as mentioned by
Armin's PEP single-quoted vs. triple-quoted strings forever.
I just don't understand the pushback here at all. This is such a
nobrainer.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 13:44 -0500, Terry Reedy wrote:
On 2/27/2012 1:01 PM, Chris McDonough wrote:
On Mon, 2012-02-27 at 12:41 -0500, R. David Murray wrote:
Eh? The 2.6 version would also be u('that'). That's the whole point
of the idiom. You'll need a better counter argument than that.
The best argument is that there already exists tons and tons of Python 2
code that already does:
u'that'
Needing to change it to:
u('that')
1) Requires effort on the part of a from-Python-2-porter to service
the aesthetic and populist goal of not having an explicit
but redundant-under-Py3 literal syntax that says this is text.
This is a point, though this would be a one-time conversion by a 2to23
converter that would be part of other needed conversions, some by hand.
I presume that most 2.6 code has problems other than u'' when attempting
to run under 3.x.
2) Won't atually meet the aesthetic goal, as
it's uglier and slower under *both* Python 2 and Python 3.
Less relevant. The minor ugliness would be in dual-version code, but not
Python 3 itself.
So the populist argument remains.. it's too confusing for people who
learn Python 3 as a new language to have a redundant syntax. But we've
had such a syntax in Python 2 for years with b'', and, as mentioned by
Armin's PEP single-quoted vs. triple-quoted strings forever.
I just don't understand the pushback here at all.
For one thing, u'' does not solve the problem for 3.1 and 3.2, while u()
does. 3.2 will be around for years. For one example, it will be in the
April long-term-support release of Ubuntu. For another, PyPy is working
on a 3.2 compatible version to come out and be put into use this year.
I suspect not everyone lives and dies by OS distribution release support
policies. Many folks are both willing and capable to install a newer
Python on an older OS.
It's unfortunate that Python 3 3.3 does not have the syntax, and
people like me who have a long-term need to straddle are to blame; we
didn't provide useful feedback early enough to avoid the mistake. That
said, it seems like preventing a reintroduction of u'' literal syntax
would presume that two wrongs make a right. By our own schedule
estimate of Python 3 takeup, many people won't be even thinking about
porting any Python 2 code to 3 until years from now.
This is such a nobrainer.
I could claim that a solution that also works for 3.1 and 3.2 is a
nobrainer. It depends on how one weighs different factors.
An argument for the reintroduction of u'' literal syntax in Python =
3.3 is not necessarily an argument against the utility of some automated
tool conversion support for porting a Python 2 app to a function-based
u() syntax so it can run in Python 3 3.2.
Tools like 2to23 or whatever can obviously be parameterized to emit
slightly different 3.2-compatible and 3.3-compatible code. It's almost
certain that it will need forward-version-aware modes like this anyway
as newer idioms are added to 3.X that make code prettier or more
efficient completely independent of u'' support.
Currently we handle 3.2 compatibility in packages that straddle via
six-like functions. We can continue doing this as necessary. If the
stdlib tooling helps, great. In an emit-function-based-syntax mode, the
conversion code would almost certainly need to rely on the import of an
externally downloadable module like six, for compatibility under both
Python 2 and 3 because there's no opportunity to go back in time and
make u() available for older releases unless it was like inlined in
every module during the conversion.
But if somebody only wants to target 3.3+, and it means they don't have
to rely on a six-like module to provide u(), great.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 15:23 -0500, R. David Murray wrote: On Mon, 27 Feb 2012 14:50:21 -0500, Chris McDonough chr...@plope.com wrote: Currently we handle 3.2 compatibility in packages that straddle via six-like functions. We can continue doing this as necessary. If the It seems to me that this undermines your argument in favor of u''. Why can't you just continue to do the above for 3.3 and beyond? I really don't know how long I'll need to do future development in the subset language of Python 2 and Python 3 because I can't predict the future. It could be two years, it might be five. Who knows. But I do know that I'm going to be developing in the subset of Python that currently runs on Python 2 = 2.6 and Python 3 = 3.2 for at least a year. And that will suck, because that language is a much less fun language in which to develop than either Python 2 or Python 3. Frankly, it's a pretty bad language. If we make this change now, it means a year from now I'll be able to develop in a slightly less sucky subset language if I choose to drop support for 3.2. And people who don't try to support Python 3 at all til then will never have to program in the suckiest subset like I will have had to. Note that u'' literals are sort of the tip of the iceberg here; supporting them will obviously not make development under the subset an order of magnitude less sucky, just a tiny little bit less sucky. There are other extremely annoying things, like str(bytes) returning the repr of a bytestring on Python 3. That's almost as irritating as the absence of u'' literals, but we have to evaluate one thing at a time. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 20:18 +, Vinay Sajip wrote:
Chris McDonough chrism at plope.com writes:
I suspect not everyone lives and dies by OS distribution release support
policies. Many folks are both willing and capable to install a newer
Python on an older OS.
But many folks aren't, and lament the slow pace of Python version adoption on
e.g. Red Hat and CentOS.
It's great to have software that installs easily. That said, the
versions of Python that my software supports is (and has to be) be my
choice.
As far as I can tell, there are maybe three or four people (besides me)
using my software on Python 3 right now. They have it pretty rough:
lackluster library support and they have to constantly mentally
transliterate third-party example code to code that works under Python
3. They are troopers!
None of them would so much as bat an eyelash if I told them today they
had to use Python 3.3 (if it existed in a final released form anyway) to
use my software. It's just a minor drop in the bucket of inconvenience
they have to currently withstand.
It's unfortunate that Python 3 3.3 does not have the syntax, and
people like me who have a long-term need to straddle are to blame; we
didn't provide useful feedback early enough to avoid the mistake. That
said, it seems like preventing a reintroduction of u'' literal syntax
would presume that two wrongs make a right. By our own schedule
estimate of Python 3 takeup, many people won't be even thinking about
porting any Python 2 code to 3 until years from now.
If the lack of u'' literal is what's holding them back, that's germane to the
discussion of the PEP. If it's not, then why propose the PEP?
Like I said in an earlier email, u'' literal support is by no means the
only issue for people who want to straddle. But it *is* an issue, and
it's incredibly low-hanging fruit with near-zero real-world impact if it
is reintroduced.
An argument for the reintroduction of u'' literal syntax in Python =
3.3 is not necessarily an argument against the utility of some automated
tool conversion support for porting a Python 2 app to a function-based
u() syntax so it can run in Python 3 3.2.
I thought the argument was more about backtracking (or not) from Python 3's
design decision to use 'xxx' for text and b'yyy' for bytes. That's the only
wrong we're talking about for this PEP, right?
You cast it as backtracking to reintroduce the syntax, but things have
changed from when the decision to omit it was first made. Its omission
introduces pain in a world where it's expected that we don't use 2to3 to
automatically translate code at installation time.
Currently we handle 3.2 compatibility in packages that straddle via
six-like functions. We can continue doing this as necessary. If the
stdlib tooling helps, great. In an emit-function-based-syntax mode, the
conversion code would almost certainly need to rely on the import of an
externally downloadable module like six, for compatibility under both
Python 2 and 3 because there's no opportunity to go back in time and
make u() available for older releases unless it was like inlined in
every module during the conversion.
But if somebody only wants to target 3.3+, and it means they don't have
to rely on a six-like module to provide u(), great.
If you only need to straddle from 2.6 onwards, then u('') isn't an issue at
all,
right now, is it?
If you look at a piece of code as something that exists in one of the
two states ported or not-ported, sure. But code often needs to be
changed, and people of varying buy-in levels need to understand and
change such code. It's just much easier for them to assume that the
same syntax works on some versions of Python 2 and Python 3 and be done
with it rather than need to explain the introduction of a function that
only exists to paper over a syntax omission.
If you need to straddle from 2.5 downwards, there are other issues to be
addressed, like exception syntax, 'with' and so forth - so making u''
available
doesn't make the port a no-brainer. And if you bite the bullet and decide to
do
the port anyway, converting u'' to u('') won't be a problem unless you (a)
can't
use a fixer to automate the conversion or (b) the function call overhead
cannot
be borne. I'm not sure either of those objections (can't use fixer, call
overhead excessive) have been made with sufficient force (i.e., data) in the
discussion so far.
Regards,
Vinay Sajip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/lists%40plope.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 21:07 +, Paul Moore wrote: On 27 February 2012 20:39, Chris McDonough chr...@plope.com wrote: Note that u'' literals are sort of the tip of the iceberg here; supporting them will obviously not make development under the subset an order of magnitude less sucky, just a tiny little bit less sucky. There are other extremely annoying things, like str(bytes) returning the repr of a bytestring on Python 3. That's almost as irritating as the absence of u'' literals, but we have to evaluate one thing at a time. So. Am I misunderstanding here, or are you suggesting that this particular PEP doesn't help you much, but if it's accepted, it represents the thin end of the wedge for a series of subsequent PEPs suggesting fixes for a number of other extremely annoying things...? I'm sure that's not what you meant, but it's certainly what it sounded like to me! I'm way too lazy. The political wrangling is just too draining (especially over something so trivial). But I will definitely support other proposals that make it easier to straddle, sure. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 21:03 +, Vinay Sajip wrote: Chris McDonough chrism at plope.com writes: I really don't know how long I'll need to do future development in the subset language of Python 2 and Python 3 because I can't predict the future. It could be two years, it might be five. Who knows. But I do know that I'm going to be developing in the subset of Python that currently runs on Python 2 = 2.6 and Python 3 = 3.2 for at least a year. And that will suck, because that language is a much less fun language in which to develop than either Python 2 or Python 3. Frankly, it's a pretty bad language. What exactly is it that makes it so bad? Since you're developing for = 2.6, what stops you from using from __future__ import unicode_literals and 'xxx' for text and b'yyy' for bytes? Then you would be working in essentially Python 3.x, at least as far as string literals go. The conversion time will be very small compared to the year time-frame you're talking about. If we make this change now, it means a year from now I'll be able to develop in a slightly less sucky subset language if I choose to drop support for 3.2. And people who don't try to support Python 3 at all til then will never have to program in the suckiest subset like I will have had to. And if we don't make the change now and you change your code to use unicode_literals, convert u'xxx' - 'xxx' and then change the places where you really meant to use bytes, that'll be a one-off change after which you will be working on a common codebase which works on 2.6+ and 3.0+, and as far as string literals are concerned you'll be working in the hopefully non-sucky 3.x syntax. Note that u'' literals are sort of the tip of the iceberg here; supporting them will obviously not make development under the subset an order of magnitude less sucky, just a tiny little bit less sucky. There are other extremely annoying things, like str(bytes) returning the repr of a bytestring on Python 3. That's almost as irritating as the absence of u'' literals, but we have to evaluate one thing at a time. Yes, but making a backward step like reintroducing u'' just to make things a tiny little bit sucky doesn't seem to me to be worth it, because then = 3.3 is different to 3.2 and earlier. Armin's suggestion of an install-time fixer is analogous to running 2to3 after every change, if you're trying to support 3.2 and 3.3+ at the same time, isn't it? You can't just edit-and-test, which to me is the main benefit of a single codebase. The downsides of a unicode_literals future import are spelled out in the PEP: http://www.python.org/dev/peps/pep-0414/#rationale-and-goals - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414 - Unicode Literals for Python 3
On Mon, 2012-02-27 at 21:43 +, Vinay Sajip wrote:
Chris McDonough chrism at plope.com writes:
It's great to have software that installs easily. That said, the
versions of Python that my software supports is (and has to be) be my
choice.
Of course. And if I understand correctly, that's 2.6, 2.7, 3.2 and later
versions. I'll ignore 2.5 and earlier in this specific reply.
None of them would so much as bat an eyelash if I told them today they
had to use Python 3.3 (if it existed in a final released form anyway) to
use my software. It's just a minor drop in the bucket of inconvenience
they have to currently withstand.
Their pain (lacklustre library support and transliterating examples from 2.x
to
3.x) would be the same under 3.2 and 3.3 (unless for some perverse reason
people
only made libraries work under one of 3.2 and 3.3, but not both).
If I had it to do all over again and a Python 3.X with unicode literals
had been available, I might not have targeted Python 3.2 at all. I
don't consider that perverse, I just consider it Python 3 water under
the bridge. Python 3.0 and 3.1 were this for me; I paid almost no
attention to them at all. Python 3.2 will be that thing for many other
people.
Like I said in an earlier email, u'' literal support is by no means the
only issue for people who want to straddle. But it *is* an issue, and
it's incredibly low-hanging fruit with near-zero real-world impact if it
is reintroduced.
But the implication of the PEP is that lack of u'' support is a major
hindrance
to porting, justifying the production of the PEP and this discussion. And it's
not low-hanging fruit with near-zero real-world impact if we're going to
deprecate it at some point (which Guido was talking about) - you're just
moving
the pain to a later date, unless we don't ever deprecate.
I personally see no need to deprecate. I can't conceive of an actual
downside to eternal backwards compatibility here. All the arguments for
its omission presume that there's some enormous untapped market full of
people yearning for its omission who would be either horrified to see
u'' or whom would not understand it on some fundamental level. I don't
think such a market actually exists. However, there *is* a huge market
for people who already understand it instinctively.
I feel, like some others, that 'xxx' is natural for text, u'xxx' is inelegant
by
comparison, and u('xxx') a little more inelegant still.
Yes, the aesthetics argument seems to be the remaining argument. I have
no problem with the aesthetics of u'' myself. But I have no problem
with the aesthetics of u('') for that matter either; if it had been used
as the prevailing style to declare something being text in Python 2 and
it had been omitted I'd be arguing for that instead. But it wasn't, of
course.
Anyway. I think I'm done doing the respond-point-for-point thing; it's
becoming diminishing returns.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Sun, 2012-02-26 at 16:06 -0500, Barry Warsaw wrote: On Feb 26, 2012, at 09:20 PM, Nick Coghlan wrote: It reduces the problem (compared to omitting the import and using a u() function), but it's still ugly and still involves the action at a distance of the unicode literals import. Frankly, that doesn't bother me at all. I've been using the future import in all my code pretty successfully for a long while now. It's much more important for a project to use or not use the future import consistently, and then there really should be no confusion when looking at the code for that project. That's completely reasonable in a highly controlled project with relatively few highly-bought-in contributors. In projects with lots of hit-and-run contributors, though, it's more desirable to have things meet a rule of least surprise. Much of the software I work on is Python 3 compatible, but it's still used primarily on Python 2. Because most people still care primarily about Python 2, and most don't have a lot of Python 3 experience, it's extremely common to see folks submitting patches with u'' literals in them. I'm not necessarily saying I'm opposed to the purpose of the PEP. I do think it's unnecessary for most Python problem domains, but can appreciate that WSGI apps are feeling a special pain here that should be addressed somehow. It would be nice however if the solution were in the form of a separate module that could be used in earlier Python versions. If we use the unicode_literals future import, or some other exernal module strategy, it doesn't help much with the hitnrun contributor thing, I fear. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 414
On Sun, 2012-02-26 at 23:06 +, Vinay Sajip wrote: Chris McDonough chrism at plope.com writes: If we use the unicode_literals future import, or some other exernal module strategy, it doesn't help much with the hitnrun contributor thing, I fear. Surely some curating of hit-and-run contributions takes place? If you accept contributions from hit-and-run contributors without changes, ISTM that could compromise the quality of the codebase somewhat. Nah. Real developers just accept all pull requests and let god sort it out. ;-) But seriously, the less time it takes me to review and fix a pull request from a casual contributor, the better. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Sat, 2011-12-10 at 15:55 +1000, Nick Coghlan wrote: So I'm back to being -1 on the idea of adding back u'' literals for 3.3. Instead, people should explicitly call str() on any literals that they want to be actual str instances both in 3.x and in 2.x when the unicode literals future import is in effect. After thinking on it a while, I can't see anything wrong with this strategy except for the 10X performance hit for defining native literals. Truth be told, in the vast majority of WSGI apps only high-level WSGI libraries (like WebOb and Werkzeug) and standalone middleware really needs to work with native strings. And the middleware really should be using the high-level libraries to parse WSGI anyway. So there are a finite number of places where it's actually a real issue. As someone who ported WebOb and other stuff built on top of it to Python 3 without using from __future__ import unicode_literals, I'm kinda sad that to be using best practice I'll have to go back and flip the polarity on everything. It's my cross to bear, though. If I have any issue with it in the future I'll bring u'' back up. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Mon, 2011-12-12 at 09:50 -0500, PJ Eby wrote: As someone who ported WebOb and other stuff built on top of it to Python 3 without using from __future__ import unicode_literals, I'm kinda sad that to be using best practice I'll have to go back and flip the polarity on everything. Eh? If you don't need unicode_literals, what's the problem? Porting the WebOb code sucked. It's only about 5K lines of code but the porting effort took me about 80 hours. Some of the problem is certainly my own idiocy, but some of it is just because straddling code across Python 2 and Python 3 currently requires that you change lots and lots of code for suspect benefit. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 22:34 -0500, Barry Warsaw wrote: On Dec 09, 2011, at 03:50 AM, Lennart Regebro wrote: One reason is that you need to be able to say This should be str in Python 2, and binary in Python 3, that should be Unicode in Python 2 and str in Python 3, and that over there should be str in both versions, and the future import doesn't support that. Sorry, I don't understand this. What does it mean to be str in both versions? And why would you want that? As for str in Python 2 and binary in Python 3, b'' prefixes do that in Python = 2.6 without the future import (if I take binary to mean bytes type). As for Unicode in Python 2 and str in Python 3, unadorned strings with the future import in Python = 2.6 does that just fine. One of the nice things too is that with #include bytesobject.h in Python = 2.6, changing all your PyStrings to PyBytes, you can get the same behavior in your extension modules. You still need to be clear about what are bytes and what are strings. The problem comes when you aren't or can't be sure, i.e. you have objects that are sometimes one and sometimes the other. Such as email headers. In that case, you're kind of screwed. Python 2's str type let you cheat, but not without consequences. Those consequences are spelled UnicodeErrors and I'll be glad to be rid of them. The PEP WSGI protocol *requires* that you present its APIs with native strings (str on Python 3, str on Python 2). So while the oversimplification don't do that sounds great here, in real life, not so much. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Fri, 2011-12-09 at 03:50 +0100, Lennart Regebro wrote: from future import unicode_literals is my fault. I'm sorry. It's pretty useless. It was suggested by somebody and I then supported it's adding, instead of allowing u'' which I suggested. But it doesn't work. One reason is that you need to be able to say This should be str in Python 2, and binary in Python 3, that should be Unicode in Python 2 and str in Python 3, and that over there should be str in both versions, and the future import doesn't support that. This is also true. But even so, b'' exists as a porting nicety. The argument for supporting u'' is the same one the one which exists for b'', except in the opposite direction. Since popular library code is going to need to run on both Python 2 and Python 3 for the foreseeable future, anything to make this easier helps. Supporting u'' in 3.3 will prevent me from needing to think about bytes/text distinction again while porting/straddling. Every time I say this to somebody who isn't listening closely they say AHA! You're *supposed* to think about bytes vs. text, that's the whole point stupid! They fail to hear the again in that sentence. I've clearly already thought about the distinction between bytes and text at least once: that's *why* I'm using a u'' literal there. I shouldn't have to think about it again to service syntax constraints. Code that is more explicit than strictly necessary should not be needlessly punished. Continuing to not support u'' in Python 3 will be like having an immigration station where folks who have a b'ritish' passport can get through right away, but folks with a u'kranian' passport need to get back on a plane that appears to come from the Ukraine before they receive another tag that says they are indeed from the Ukraine. It's just pointless makework. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 19:52 -0500, Glyph wrote: Zooming back in to the actual issue this thread is about, I think the u-vs- issue is a bit of a red herring, because the _real_ problem here is that 2to3 is slow and buggy and so migration efforts are starting to work around it, and therefore want to run the same code on 3.x and all the way back to 2.5. Even if it weren't slow, I still wouldn't use it to automatically convert code at install time; a single codebase is easier to reason about, and easier to support. Users send me tracebacks all the time; having them match the source is a wonderful thing. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 21:43 -0800, Guido van Rossum wrote:
On Thu, Dec 8, 2011 at 9:33 PM, Chris McDonough chr...@plope.com
wrote:
On Thu, 2011-12-08 at 19:52 -0500, Glyph wrote:
Zooming back in to the actual issue this thread is about, I
think the
u-vs- issue is a bit of a red herring, because the
_real_ problem
here is that 2to3 is slow and buggy and so migration efforts
are
starting to work around it, and therefore want to run the
same code on
3.x and all the way back to 2.5.
Even if it weren't slow, I still wouldn't use it to
automatically
convert code at install time; a single codebase is easier to
reason
about, and easier to support. Users send me tracebacks all
the time;
having them match the source is a wonderful thing.
Even though 2to3 was my idea, I am gradually beginning to appreciate
this approach. I skimmed the docs for six and liked it.
But I think the specific proposal of adding u... literals back to
3.3 is not going to do much good. If we had had the foresight way back
when, we could have added them back to 3.1 and we would have been
okay. But having them in 3.3 but not in 3.2 is just adding insult to
injury.
AFAICT, at the current pace of porting, lots of authors of existing,
popular Python 2 libraries won't be releasing a ported/straddled version
any time soon; almost certainly many won't even begin work on a port
until after 3.3 is final. As a result, on the supplier side, there will
be plenty of code that will eventually work only as a straddle across
2.6, 2.7, and 3.3.
On the consumer side, folks who want to run 2.6/2.7/3.3-only codebases
will have the wherewithal to compile their own Python 3 (or use a PPA or
equivalent) until the distros catch up.
So I'm not sure why 3.2 not having support for u'' should be a real
blocker for the change.
I recommend writing bdecode('utf-8'); maybe six's u() does the
same?
It does this:
def u(s):
return unicode(s, unicode_escape)
That's two Python function calls, of course, which is obviously icky if
you use a lot of literals at a nonmodule scope.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Fri, 2011-12-09 at 16:36 +1000, Nick Coghlan wrote: On Fri, Dec 9, 2011 at 4:01 PM, Chris McDonough chr...@plope.com wrote: On the consumer side, folks who want to run 2.6/2.7/3.3-only codebases will have the wherewithal to compile their own Python 3 (or use a PPA or equivalent) until the distros catch up. So I'm not sure why 3.2 not having support for u'' should be a real blocker for the change. If this argument was valid, people wouldn't be so worried about maintaining 2.5 compatibility in their libraries. Consider if I tried to make this argument to justify everyone dropping 2.5 and earlier support today: On the consumer side, folks who want to run 2.6+ codebases on older Linux distros have the wherewithal to compile their own more recent Python 2 (or use a PPA or equivalent) until they can move to a more recent version of their distro. Fair point. That said, personally, I have given up entirely on Python 2.4 and 2.5 support for newer versions of my OSS libraries. I continue to backport fixes and (some) features to older library versions so folks can run those on systems that require older Pythons. I gave up 2.5 support fairly recently across everything new, and I gave up support for 2.4 a year ago or more in new releases with the same intent. In reality, there is only one major platform that requires 2.4: RHEL 5 and folks who use it will just need to also use old versions of popular libraries; trying to support it for all future feature work until it's EOLed is not sane unless someone pays for it. Python 2.5 has slightly more compelling platforms (GAE and Jython), but GAE is moving to Python 2.7 and Jython is a bit moribund these days and is not really popular enough that a critical mass of folks will clamor for new-and-shiny releases that run on it. The upshot is that most newly created code only needs to run on Python 2.6 and *some* version of Python 3. And being able to eventually write that code in a nonsucky subset of Python 2/3 is important to me, because I'm going to be developing software in that subset for many years (way past the timeframe we're talking about in which Python 3.2 will rule the roost). It's simply not true in the general case - people don't maintain 2.4+ compatibility for fun, they do it because RHEL5 (and CentOS 5, etc) are still reasonably common and ship with 2.4 as the system Python. As soon as you switch away from the system provided Python, you're switching away from the vendors entire pre-packaged Python *stack*, not just the interpreter itself. You then have to install (and generally build) *everything* for yourself. While that is certainly possible these days (and a lot simpler than it used to be), it's still not trivial [1]. Since 3.2 is already quite usable for applications that aren't fighting with the native strings problem (which seems to be the common thread running through the complaints I've heard from web framework authors), and with it being included in at least the next Ubuntu LTS, current versions of Fedora, Arch, etc, it's going to be around for a long time. Ignoring 3.1 is a reasonable option. Ignoring 3.2 entirely is unlikely to be viable for anyone that is interested in supporting 3.x within the next couple of years - the 3.3 release is at least 9 months away, and it's also going to take a while for it to make its way into distros after the final release gets published on python.org. Hence my suggestion: perhaps the problem is the fact that PEP 3.3/WSGI 1.0.1 introduced the native string concept as a minimalist hack to try to get a usable gateway interface in Python 3, and that just doesn't work in practice when attempting to straddle 2.x and 3.x (because the values WSGI is dealing with aren't really text, they're bytes, only *some* of which represent text). Perhaps a PEP 444 based model would be less painful and more coherent in the long run? Possibly. I was the original author of PEP 444 with help from Armin. (although it has since been taken up by Alice and I do not support the updates it has received since then). A bytes-oriented WSGI-like protocol was always the saner option. The native string idea optimized in exactly the wrong place, which was to make it easy to write WSGI middleware, where you're required to do lots of textlike manipulation of header values. The idea of using bytes in places where PEP now mandates native strings was rejected because people were (somewhat justifiably) horrified at what they had to do in order to attempt treat bytes like strings in this context on Python 3 at the time. It has gotten better, but maybe still not better enough to appease the folks who blocked the idea originally. But all of that is just arguing with the umpire at this point. Promoting and getting consensus about a different protocol will hurt a lot. PEP was borne of months of intense periods of arguing and compromise. It is the way it is now because everyone
[Python-Dev] readd u'' literal support in 3.3?
On the heels of Armin's blog post about the troubles of making the same
codebase run on both Python 2 and Python 3, I have a concrete
suggestion.
It would help a lot for code that straddles both Py2 and Py3 to be able
to make use of u'' literals. It would seem to be an easy thing to
reenable (see
http://www.reddit.com/r/Python/comments/n3q7q/thoughts_on_python_3_armin_ronachers_thoughts_and/c36397t
) . It would seem to cost very little in terms of maintenance, and not much
in docs.
It would make it possible to share code like this across py2 and py3:
a = u'foo'
Instead of (with e.g. six):
a = u('foo')
Or:
from __future__ import unicode_literals
a = 'foo'
I recognize that the last option is probably the way its meant to be
done, but in reality it's just more practical to not fail when literal
notation is more specific than strictly necessary.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 01:02 -0500, Benjamin Peterson wrote: 2011/12/8 Chris McDonough chr...@plope.com: On the heels of Armin's blog post about the troubles of making the same codebase run on both Python 2 and Python 3, I have a concrete suggestion. It would help a lot for code that straddles both Py2 and Py3 to be able to make use of u'' literals. Helpful or not helpful, I think that ship has sailed. The earliest it could see the light of day is 3.3, which would leave people trying to support 3.1 and 3.2 in a bind. Right.. the title does say readd ... support in 3.3. Are you suggesting the ship has sailed for eternity because it can't be supported in Python 3.3? - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 01:18 -0500, Benjamin Peterson wrote: 2011/12/8 Chris McDonough chr...@plope.com: On Thu, 2011-12-08 at 01:02 -0500, Benjamin Peterson wrote: 2011/12/8 Chris McDonough chr...@plope.com: On the heels of Armin's blog post about the troubles of making the same codebase run on both Python 2 and Python 3, I have a concrete suggestion. It would help a lot for code that straddles both Py2 and Py3 to be able to make use of u'' literals. Helpful or not helpful, I think that ship has sailed. The earliest it could see the light of day is 3.3, which would leave people trying to support 3.1 and 3.2 in a bind. Right.. the title does say readd ... support in 3.3. Are you suggesting the ship has sailed for eternity because it can't be supported in Python 3.3? I'm questioning the real utility of it. All I can really offer is my own experience here based on writing code that needs to straddle Python 2.5, 2.6, 2.7 and 3.2 without use of 2to3. Having u'' work across all of these would mean porting would not require as much eyeballing as code modified via from future import unicode_literals, it would let more code work on 2.5 unchanged, and the resulting code would execute faster than code that required us to use a u() function. What's the case against? - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] readd u'' literal support in 3.3?
On Thu, 2011-12-08 at 17:33 +1000, Nick Coghlan wrote: Such code still won't work on 3.2, hence restoring the redundant notation would be ultimately pointless. None of the code I've written which straddles Python 2/3 supports anything except Python 3.2+, and likewise I expect that for the next crop of porters/straddlers, their code won't support anything but Python 3.3+. So there is a point, which is to make it easier for people to port code that can straddle the most recent Python 3 release as well as 2.7/2.6. In that context, I don't see much relevance of having no support for u'' in Python 3.2. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
This is really very comprehensive, thank you! Why not modify sys.prefix? - -- As discussed above under `Backwards Compatibility`_, this PEP proposes to add ``sys.site_prefix`` as the prefix relative to which site-package directories are found. This maintains compatibility with the documented meaning of ``sys.prefix`` (as the location relative to which the standard library can be found), but means that code assuming that site-packages directories are found relative to ``sys.prefix`` will not respect the virtual environment correctly. Since it is unable to modify ``distutils``/``sysconfig``, `virtualenv`_ is forced to instead re-point ``sys.prefix`` at the virtual environment. An argument could be made that this PEP should follow virtualenv's lead here (and introduce something like ``sys.base_prefix`` to point to the standard library and header files), since virtualenv already does this and it doesn't appear to have caused major problems with existing code. Another argument in favor of this is that it would be preferable to err on the side of greater, rather than lesser, isolation. Changing ``sys.prefix`` to point to the virtual environment and introducing a new ``sys.base_prefix`` attribute would err on the side of greater isolation in the face of existing code's use of ``sys.prefix``. It would seem to make sense to me to err on the side of greater isolation, introducing sys.base_prefix to indicate the base prefix (as opposed to sys.site_prefix indicating the venv prefix). Bugs introduced via a semi-isolated virtual environment are very difficult to troubleshoot. It would also make changes to existing code unnecessary. I have encountered no issues with virtualenv doing this so far. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging in Python 2 anyone ?
I'll throw this out there.. why is it going to have a different name on python2 than on python3? - C On Wed, 2011-08-17 at 22:30 +, Vinay Sajip wrote: Tarek Ziadé ziade.tarek at gmail.com writes: IOW, the task to do is: 1/ copy packaging and all its stdlib dependencies in a standalone project 2/ rename packaging to distutils2 3/ make it work under older 2.x and 3.x (2.x would be the priority) 4/ release it, promote its usage 5/ consolidate the API with the feedback received I realize it's by far the less interesting task to do in packaging, but it's by far one of the most important Okay, I had a bit of spare time today, and here's as far as I've got: Step 1 - done. Step 2 - done. Step 3 - On Python 2.6 most of the tests pass: Ran 322 tests in 12.148s FAILED (failures=3, errors=4, skipped=39) See the detailed test results (for Linux) at https://gist.github.com/1152791 The code is at https://bitbucket.org/vinay.sajip/du2/ stdlib dependency code is either moved to util.py or test/support.py as appropriate. You need to test in a virtualenv with unittest2 installed. No work has been done on packaging the project. I'm not sure if I'll have much more time to spend on this, but it's there in case someone else can look at the remaining test failures, plus Steps 4 and 5; hopefully I've broken the back of it :-) Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Tue, 2010-09-21 at 23:38 +1000, Nick Coghlan wrote: And if this turns out to be a disaster in practice: a) on my head be it; and b) we still have the option of the DeprecationWarning dance for bytes inputs to the existing functions and moving to a parallel API In the case of urllib.parse, it's entirely safe. If someone beats you up over it later, you can tell them to bother straddlers in Web-SIG, as we're the folks who most want the polymorphism in that particular API. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3
On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: While the Web-SIG is trying to hash out PEP 444, I thought it would be a good idea to have a backup plan that would allow the Python 3 stdlib to move forward, without needing a major new spec to settle out implementation questions. If a WSGI-1-compatible protocol seems more sensible to folks, I'm personally happy to defer discussion on PEP 444 or any other backwards-incompatible proposal. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Sun, 2010-09-19 at 12:03 +1000, Nick Coghlan wrote: On Sun, Sep 19, 2010 at 4:18 AM, John Nagle na...@animats.com wrote: On 9/18/2010 2:29 AM, python-dev-requ...@python.org wrote: Polymorphic best practices [was: (Not) delaying the 3.2 release] If you're hung up on this, try writing the user-level documentation first. Your target audience is a working-level Web programmer, not someone who knows six programming languages and has a CS degree. If the explanation is too complex, so is the design. Coding in this area is quite hard to do right. There are issues with character set, HTML encoding, URL encoding, and internationalized domain names. It's often done wrong; I recently found a Google service which botched it. Python libraries should strive to deliver textual data to the programmer in clean Unicode. If someone needs the underlying wire representation it should be available, but not the default. Even though URL byte sequences are defined as using only an ASCII subset, I'm currently inclined to add raw bytes supports to urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, etc) rather than doing it implicitly in the normal functions. My rationale is as follows: - while URLs are *meant* to be encoded correctly as an ASCII subset, the real world isn't always quite so tidy (i.e. applications treat as URLs things that technically are not because the encoding is wrong) - separating the APIs forces the programmer to declare that they know they're working with the raw bytes off the wire to avoid the decode/encode overhead that comes with working in the Unicode domain - easier to change our minds later. Adding implicit bytes support to the normal names can be done any time, but removing it would require an extensive deprecation period Essentially, while I can see strong use cases for wanting to manipulate URLs in wire format, I *don't* see strong use cases for manipulating URLs without *knowing* whether they're in wire format (encoded bytes) or display format (Unicode text). For some APIs that work for arbitrary encodings (e.g. os.listdir) switching based on argument type seems like a reasonable idea. For those that may silently produce incorrect output for ASCII-incompatible encodings, the os.environ/os.environb seems like a better approach. urllib.parse.urlparse/urllib.parse.urlsplit will never need to decode anything when passed bytes input. Both could just put the bytes comprising the hex-encoded components (the path and query string) into its respective place in the parse results, just like it does now for string input. As far as I can tell, the only thing preventing it from working against bytes right now is the use of string literals in the source instead of input-type-dictated-literals. There should not really be any need to create a urllib.parse.urlsplitb unless the goal is to continue down the (not great IMO) precedent already set by the shadow bytes API in urllib.parse (*_to_bytes, *_from_bytes) or if we just want to make it deliberately harder to parse URLs. The only decoding that needs to be done to potential bytes input by APIs in urllib.parse will be in the face of percent encodings in the path and query components (handled entirely by unquote and unquote_plus, which already deal in bytes under the hood). The only encoding that needs to be done by urllib.parse is in the face of input to the urlencode and quote APIs. quote already deals with bytes as input under the hood. urlencode does not, but it might be changed use the same strategy that quote does now (by using a urlencode_to_bytes under the hood). However, I think any thought about adding raw bytes support is largely moot at this point. This pool has already been peed in.There's effectively already a shadow bytes-only API in the urlparse module in the form of the *_to_bytes and *_from_bytes functions in most places where it counts. So as I see it, the options are: 1) continue the *_to_bytes and *_from_bytes pattern as necessary. 2) create a new module (urllib.parse2) that has only polymorphic functions. #1 is not very pleasant to think about as a web developer if I need to maintain a both-2-and-3-compatible codebase. Neither is #2, really, if I had to support Python 3.1 and 3.2. From my (obviously limited) perspective, a more attractive third option is backwards incompatibility in a later Python 3 version, where encoding-aware functions like quote, urlencode, and unquote_plus were polymorphic, accepting both bytes and string objects and returning same-typed data. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Mon, 2010-09-20 at 23:23 +1000, Nick Coghlan wrote:
On Mon, Sep 20, 2010 at 10:12 PM, Chris McDonough chr...@plope.com wrote:
urllib.parse.urlparse/urllib.parse.urlsplit will never need to decode
anything when passed bytes input.
Correct. Supporting manipulation of bytes directly is primarily a
speed hack for when an application wants to avoid the
decoding/encoding overhead needed to perform the operations in the
text domain when the fragments being manipulated are all already
correctly encoded ASCII text.
The urllib.parse.urlparse/urlsplit functions should never need to know
or care whether the input they're passed is correctly encoded. They
actually don't care right now: both will happily consume non-ASCII
characters and spit out nonsense in the parse results. If passed
garbage, they'll return garbage:
urlparse('http://www.cwi.nl:80/%7Eguido/LaPeña.html')
ParseResult(scheme='http', netloc='www.cwi.nl:80',
path='/%7Eguido/LaPeña.html', params='',
query='', fragment='')
However, supporting direct manipulation of bytes *implicitly* in the
current functions is problematic, since it means that the function may
fail silently when given bytes that are encoded with an ASCII
incompatible codec (or which there are many, especially when it comes
to multibyte codecs and other stateful codecs). Even ASCII compatible
codecs are a potential source of hard to detect bugs, since using
different encodings for different fragments will lead directly to
mojibake.
The path component result above is potentially useless and broken, and
if it is inserted into a web page as a link, it may cause mojibake, but
urlparse doesn't (can't) complain. As far as I can tell, there would be
no more and no less potential for mojibake if the same API were able to
be fed a bytes object. The result can already be nonsense and allowing
for bytes as input doesn't add any greater potential to receive nonsense
back.
Most APIs in urllib.parse exhibit the same behavior today, e.g. urljoin:
urljoin('http://gooñle.com', '%7Eguido/LaPeña.html')
'http://gooñle.com/%7Eguido/LaPeña.html'
The resulting URL is total nonsense.
Moving the raw bytes support out to separate APIs allows their
constraints to be spelled out clearly and for programmers to make a
conscious decision that that is what they want to do. The onus is then
on the programmer to get their encodings correct.
I guess my argument is that the onus already *is* on the programmer to
get their encodings right. They can just as easily screw up while using
str inputs.
If we decide to add implicit support later, that's pretty easy (just
have urllib.parse.* delegate to urllib.parse.*b when given bytes).
Taking implicit support *away* after providing it, however, means
going through the whole deprecation song and dance. Given the choice,
I prefer the API design that allows me to more easily change my mind
later if I decide I made the wrong call.
Existing APIs save for quote don't really need to deal with charset
encodings at all, at least on any level that Python needs to care about.
The potential already exists to emit garbage which will turn into
mojibake from almost all existing APIs. The only remaining issue seems
to be fear of making a design mistake while designing APIs.
IMO, having a separate module for all urllib.parse APIs, each designed
for only bytes input is a design mistake greater than any mistake that
could be made by allowing for both bytes and str input to existing APIs
and returning whatever type was passed. The existence of such a module
will make it more difficult to maintain a codebase which straddles
Python 2 and Python 3.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Tue, 2010-09-21 at 07:12 +1000, Nick Coghlan wrote: On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough chr...@plope.com wrote: Existing APIs save for quote don't really need to deal with charset encodings at all, at least on any level that Python needs to care about. The potential already exists to emit garbage which will turn into mojibake from almost all existing APIs. The only remaining issue seems to be fear of making a design mistake while designing APIs. IMO, having a separate module for all urllib.parse APIs, each designed for only bytes input is a design mistake greater than any mistake that could be made by allowing for both bytes and str input to existing APIs and returning whatever type was passed. The existence of such a module will make it more difficult to maintain a codebase which straddles Python 2 and Python 3. Failure to use quote/unquote correctly is a completely different problem from using bytes with an ASCII incompatible encoding, or mixing bytes with different encodings. Yes, if you don't quote your URLs you may end up with mojibake. That's not a justification for creating a *new* way to accidentally create mojibake. There's no new way to accidentally create new mojibake here by allowing bytes input, as far as I can tell. - If a user passes something that has character data outside the range 0-127 to an API that expects a URL or a component (in the definition that urllib.parse.urlparse uses for component) of a URI, he can keep both pieces when it breaks. Whether that data is represented via bytes or text is not relevant. He provided bad input, he is going to lose one way or another. - If a user passes a bytestring to ``quote``, because ``quote`` is implemented in terms of ``quote_to_bytes`` the case is *already* handled by quote_to_bytes implicitly failing to convert nonascii characters. What are the cases you believe will cause new mojibake? Separating the APIs means that application programmers will be expected to know whether they are working with data formatted for display to the user (i.e. Unicode text) or transfer over the wire (i.e. ASCII compatible bytes). Can you give me a concrete use case where the application programmer won't *know* which format they're working with? Py3k made the conscious decision to stop allowing careless mixing of encoded and unencoded text. This is just taking that philosophy and propagating it further up the API stack (as has already been done with several OS facing APIs for 3.2). Yes. Code which must explicitly deal with bytes input and output meant to straddle both Python 2 and Python 3. Please try to write some code which 1) uses the same codebase to straddle Python 2.6 and Python 3.2 and 2) which uses bytes input, and expects bytes output from, say, urlsplit. It becomes complex very quickly. A proposal to create yet another bytes-only API only makes it more complex, AFAICT. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
On Tue, 2010-09-21 at 08:19 +1000, Nick Coghlan wrote: On Tue, Sep 21, 2010 at 7:39 AM, Chris McDonough chr...@plope.com wrote: On Tue, 2010-09-21 at 07:12 +1000, Nick Coghlan wrote: On Tue, Sep 21, 2010 at 4:30 AM, Chris McDonough chr...@plope.com wrote: Existing APIs save for quote don't really need to deal with charset encodings at all, at least on any level that Python needs to care about. The potential already exists to emit garbage which will turn into mojibake from almost all existing APIs. The only remaining issue seems to be fear of making a design mistake while designing APIs. IMO, having a separate module for all urllib.parse APIs, each designed for only bytes input is a design mistake greater than any mistake that could be made by allowing for both bytes and str input to existing APIs and returning whatever type was passed. The existence of such a module will make it more difficult to maintain a codebase which straddles Python 2 and Python 3. Failure to use quote/unquote correctly is a completely different problem from using bytes with an ASCII incompatible encoding, or mixing bytes with different encodings. Yes, if you don't quote your URLs you may end up with mojibake. That's not a justification for creating a *new* way to accidentally create mojibake. There's no new way to accidentally create new mojibake here by allowing bytes input, as far as I can tell. - If a user passes something that has character data outside the range 0-127 to an API that expects a URL or a component (in the definition that urllib.parse.urlparse uses for component) of a URI, he can keep both pieces when it breaks. Whether that data is represented via bytes or text is not relevant. He provided bad input, he is going to lose one way or another. - If a user passes a bytestring to ``quote``, because ``quote`` is implemented in terms of ``quote_to_bytes`` the case is *already* handled by quote_to_bytes implicitly failing to convert nonascii characters. What are the cases you believe will cause new mojibake? Calling operations like urlsplit on byte sequences in non-ASCII compatible encodings and operations like urljoin on byte sequences that are encoded with different encodings. These errors differ from the URL escaping errors you cite, since they can produce true mojibake (i.e. a byte sequence without a single consistent encoding), rather than merely non-compliant URLs. However, if someone has let their encodings get that badly out of whack in URL manipulation they're probably doomed anyway... Right, the bytes issue here is really a red herring in both the urlsplit and urljoin cases, I think. It's certainly possible I hadn't given enough weight to the practical issues associated with migration of existing code from 2.x to 3.x (particularly with the precedent of some degree of polymorphism being set back when Issue 3300 was dealt with). Given that a separate API still places the onus on the developer to manage their encodings correctly, I'm beginning to lean back towards the idea of a polymorphic API rather than separate functions. (the quote/unquote legacy becomes somewhat unfortunate in that situation, as they always returns str objects rather than allowing the type of the result to be determined by the type of the argument. Something like quotep/unquotep may prove necessary in order to work around that situation and provide a bytes-bytes, str-str API) Yay, sounds much, much better! - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.x as the official release
For reference, I have developed a spec and an (untested) reference implementation of a WSGI successor I've given the name Web3. Ian is not hot on this spec (he prefers native strings as environ keys). I'm definitely not going to write a WebOb analogue, so I'd more or less given up trying to promote it. But it's here for consideration. Given that nobody else has written this all out in spec form, it may be useful: http://github.com/mcdonc/web3 - C On Wed, 2010-09-15 at 11:11 -0700, Guido van Rossum wrote: Given that wsgiref is in the stdlib, I think we should hold up the 3.2 release (and even the first beta) until this is resolved, unless we can convince ourselves that it's okay to delete wsgiref from the stdlib (which sounds unlikely but may not be any more incompatible than making it work properly :-). I want to emphasize that I am *not* a stakeholder so my preference for bytes or Unicode shouldn't matter; that said, given WSGI's traditional emphasis on using the lowest-level, vanilla standard datatypes (e.g. you can't even subclass dict let alone provide another kind of mapping -- it has to be a real dict) it makes sense to me that the values should be bytes, os.environ notwithstanding. The keys probably could be Unicode (HTTP headers are required to use only 7-bit ASCII characters anyways right?). But I'd be happy to be shown the error of my ways (or given a link showing prior discussion of the matter -- preferably with a conclusion :-). --Guido On Wed, Sep 15, 2010 at 10:36 AM, Jacob Kaplan-Moss ja...@jacobian.org wrote: On Wed, Sep 15, 2010 at 12:09 PM, Jesse Noller jnol...@gmail.com wrote: Fundamentally; I would gladly hold up 3.2 (just my opinion) for the needed fixes to the standard lib [...] I think I should share a little anecdote at this point: Earlier in the year I worked for a while on Django/Py3. It's actually not that hard of a task (because I'm building on the work by MvL and some of Greg Wilson's students!) and I quickly got a simple app working locally. So the next step for me was to see about putting the app into production... and that's where the wheels fell off. So that's where I stopped. As far as I'm concerned, I'm not willing to expend the effort to get Django ported if I can't put it into production. Most of us working on Django are going to feel the same way, I suspect. Further, I can say with some confidence that until the WSGI issue is sorted the Python web world isn't going to have much enthusiasm for Python 3. I'm trying not to sound overly negative here -- really, I can't *WAIT* to be able to switch to Py3! But until I have a bunch of interoperable, robust WSGI servers like I do on Python 2 -- modwsgi, uwsgi, cherrypy, gunicorn, ... -- Python 3 is going to remain a pipe dream. Jacob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to decide on a Python 3 design for wsgiref
On Wed, 2010-09-15 at 13:46 -0700, Brett Cannon wrote: Both the RM and BDFL agree that Python 3.2b1 should be held up until we settle this wsgi matter. That makes it a question of how to settle it. Thinking out loud here to keep this discussion focused, I say we give a deadline for PEPs to be submitted by October 15th. We then choose two PEP dictators to make a call by November 1, get wsgiref cleaned up ASAP, and get Python 3.2b1 out the door immediately thereafter. If web-sig manages to come to an agreement as a whole before then we can skip the PEPs, but if they have not managed to do this already then it probably is not going to suddenly happen now under threat of python-dev making the call for them by blessing a new wsgiref implementation (happy to be proven wrong, though). FWIW, I've submitted http://github.com/mcdonc/web3/blob/master/web3.rst as a PEP (by sending a mail to p...@python.org). - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 444 aka Web3 (was Re: how to decide on a Python 3 design for wsgiref)
This spec, a WSGI-like spec for Python2 and Python3 named Web3 was accepted today as PEP 444. http://python.org/dev/peps/pep-0444/ Comments and competing specs would be useful. - C On Wed, 2010-09-15 at 18:16 -0400, Chris McDonough wrote: On Wed, 2010-09-15 at 13:46 -0700, Brett Cannon wrote: Both the RM and BDFL agree that Python 3.2b1 should be held up until we settle this wsgi matter. That makes it a question of how to settle it. Thinking out loud here to keep this discussion focused, I say we give a deadline for PEPs to be submitted by October 15th. We then choose two PEP dictators to make a call by November 1, get wsgiref cleaned up ASAP, and get Python 3.2b1 out the door immediately thereafter. If web-sig manages to come to an agreement as a whole before then we can skip the PEPs, but if they have not managed to do this already then it probably is not going to suddenly happen now under threat of python-dev making the call for them by blessing a new wsgiref implementation (happy to be proven wrong, though). FWIW, I've submitted http://github.com/mcdonc/web3/blob/master/web3.rst as a PEP (by sending a mail to p...@python.org). - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Future of 2.x.
On Wed, 2010-06-09 at 01:15 -0400, Fred Drake wrote: On Wed, Jun 9, 2010 at 12:30 AM, Senthil Kumaran orsent...@gmail.com wrote: it would still be a good idea to introduce some of them in minor releases in 2.7. I know, this deviating from the process, but it could be an option considering that 2.7 is the last of 2.x release. I disagree. If there are going to be features going into *any* post 2.7.0 version, there's no reason not to increment the revision number to 2.8, Since there's also a well-advertised decision that 2.7 will be the last 2.x, such a 2.8 isn't planned. But there's no reason to violate the no-features-in-bugfix-releases policy. We've seen violations cause trouble and confusion, but we've not seen it be successful. The policy wasn't arbitrary; let's stick to it. It might be useful to copy the identifiers and URLs of all the backport request tickets into some other repository, or to create some unique state in roundup for these. Rationale: it's almost certain that if the existing Python core maintainers won't evolve Python 2.X past 2.7, some other group will, and losing existing context for that would kinda suck. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCon Keynote
Guido van Rossum wrote: Please mail me topics you'd like to hear me talk about in my keynote at PyCon this year. Can you tell us where Uncle Timmy has been and when he'll be back? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7 alpha 2
Brett Cannon wrote: IMO keeping Python 2.x around past 2.7 is the equivalent of python-dev saying Python 3 is the future, but we are keeping the old Python 2.x around because we don't have *that* much faith in the future we have laid out. That's poison to Python 3 by showing a lack of confidence in the direction that the BDFL and python-dev as a group has chosen. Now I could be wrong and there could actually be a large number of active contributors who want to keep the 2.x series going, but based on the discussion that occurred the last time this came up I believe the guys who are keeping Python running are ready to move on. I think this is mostly just a branding issue. Once the folks who have historically kept Python 2 running really *do* move on, there will be other folks who will step up and become maintainers out of necessity: those stuck on old platforms, permanent stragglers, people who for whatever reason actually like Python 2 better, etc. It's just going to happen, there's really nothing anybody can do about it. I don't think there's anything wrong with this. If such a group of folks wants to get together and create another Python 2.X release, there's nothing stopping them from doing so except the above branding declaration. I'd personally not close the door on allowing these folks to call such an effort Python 2. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view
I've been trying to follow this discussion now for weeks. The signal to noise ratio is pretty low. I'd love to have an stdlib solution for distribution packaging and installation. But I think we might as well pack it up and go home if the folks whom are contributing to the discussion recreationally (whom are not themselves implementers and potential implementers or spec writers or potential spec writers of packaging systems) continue to chime in on *every single issue*, contributing only stop energy. It's just completely pointless. - C On 7/15/09 12:34 PM, P.J. Eby wrote: At 04:59 PM 7/15/2009 +0100, Paul Moore wrote: - Virtualenv isn't a workaround (I don't know virtualenv, I'll take your word for it) It's not one for system package maintainers because it would effectively be managing multiple instances of 'python'. Really not a suitable solution. - I do not believe that it's clear that sanctioning the setuptools workaround as the right approach by building it into the Python core/stdlib is the right thing to do. I still don't understand how we're doing that. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of medusa
Daniel Stutzbach wrote: On Wed, Mar 4, 2009 at 12:54 PM, Guido van Rossum gu...@python.org wrote: The same as always. We don't change APIs in bugfix releases. This question is actually for the Zope folks and others who have had problems with the 2.6 asyncore/asynchat: Are any of the problems due to a change in the documented API... or are they all due to changes in undocumented internals that your code relied on? As far as I can tell, asyncore/asynchat is all undocumented internals. Any use of asyncore in anger will use internals; there never was any well-understood API to these modules. Medusa itself (from which asyncore and asynchat were derived) appears to itself break with the changes to asyncore/asynchat in 2.6 (at least it appears to use attributes like ac_out_buffer which were removed in 2.6; this is not Zope's version; there is no such animal; this is plain old Medusa 0.5.4). Count me in as one who believes that it would be the lesser of two evils to revert to the older (2.5 and prior) asyncore/asynchat implementations in 2.6.2 rather than soldiering on with the 2.6 and 2.6.1 implementation (which almost certainly has fewer, if any, consumers); somebody messed up in 2.6 by making its asyncore/asynchat more forward compatible with 3.0's than backwards compatible with 2.5's and prior; this was just a mistake and keeping old code running should trump any theoretical or puritanical benefit in a dot release. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to easily consume just the parts of eggs that are good for you
zooko wrote:
On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote:
zooko wrote:
http://mail.python.org/pipermail/python-dev/2008-March/078243.html
Here is a simple proposal: make the standard Python import
mechanism notice eggs on the PYTHONPATH and insert them (into the
*same* location) on the sys.path.
This eliminates the #1 problem with eggs -- that they don't easily
work when installing them into places other than your site-packages
and that if you allow any of them to be installed on your system
then they take precedence over your non-egg packages even you
explicitly put those other packages earlier in your PYTHONPATH.
(That latter behavior is very disagreeable to more than a few
prorgammers.)
Sorry if I'm out of the loop and there's some subtlety here that I'm
disregarding, but it doesn't appear that either of the issues you
mention is a actually problem with eggs. These are instead problems
with how eggs get installed by easy_install (which uses a .pth file to
extend sys.path). It's reasonable to put eggs on the PYTHONPATH
manually (e.g. sys.path.append('/path/to/some.egg')) instead of using
easy_install to install them.
Yes, you are missing something. While many programmers, such as
yourself and Lennart Regebro (who posted to this thread) find the
current eggs system to be perfectly convenient and to Just Work, many
others, such as Glyph Lefkowitz (who posted to a related thread) find
them to be so annoying that they actively ensure that no eggs are ever
allowed to touch their system.
The reasons for this latter problem are two:
1. You can't conveniently install eggs into a non-system directory,
such as ~/my-python-stuff.
That's just not true.
$ export PYTHONPATH=/home/you/my-python-stuff/foo-1.3.egg
$ python
import foo
Eggs are directories (or zipfiles) that contain packages. They happen to
contain other metadata directories too, but these can be ignored if you just
want to *use* them (as opposed to wanting to introspect metadata about them).
2. If you allow even a single egg to be installed into your PYTHONPATH,
it will change the semantics of your PYTHONPATH.
I think you are mistaken. The use of the .pth file that changes sys.path is a
feature of easy_install, not of eggs. You don't need to use any .pth file to
put eggs on your PYTHONPATH.
Note that zc.buildout is a framework that installs eggs, and it doesn't rely at
all on .pth files to automatically hack sys.path. Instead, it munges console
scripts to add each egg dir to sys.path. This is pretty nasty too, but it does
prove the point.
It is however true that you need to change sys.path to use an egg. Is that
what
you're objecting to fundamentally? You just don't want to have to change
sys.path at all to use an egg? Maybe you're objecting to the notion that an
egg
can contain more than one package. There is functionally no difference between
an egg and a directory full of packages.
Both of these problems are directly caused by the need for eggs to hack
your site.py. If Python automatically added eggs found in the
PYTHONPATH to the sys.path, both of these problems would go away.
I'm not sure what you mean. Eggs don't hack site.py. Eggs are just a
packaging
format. easy_install doesn't hack site.py either. It just puts a .pth file
(the parsing of which is a feature of core Python itself, not any setuptools
magic) in site packages and manages it.
It seems like you're advocating adding magic that you can't turn off (magical
detection of eggs in an already site.py-approved packages directory) to defeat
magic that you can turn off (by not using easy_install or .pth files). At some
level the magic you want to see built in to Python would almost certainly wind
up doing what you hate by hacking sys.path unless you wanted to restrict eggs
to
containing a single package only. And you wouldn't be able to turn it off
except through some obscure environment variable setting.
By the way, since I posted my proposal two weeks ago I have pointed a
couple of Python hackers who currently refuse to use eggs at the URL:
http://mail.python.org/pipermail/python-dev/2008-March/078243.html
They both agreed that it made perfect sense. I told one of them about
the alternate proposal to define a new database file to contain a list
of installed packages, and he sighed and rolled his eyes and said So
they are planning to reinvent apt!.
I think changing the Python core is the worst possible answer to this problem.
Don't use easy_install is currently the best, AFAICT.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to easily consume just the parts of eggs that are good for you
zooko wrote:
Folks:
Here is a simple proposal: make the standard Python import
mechanism notice eggs on the PYTHONPATH and insert them (into the
*same* location) on the sys.path.
This eliminates the #1 problem with eggs -- that they don't easily
work when installing them into places other than your site-packages
and that if you allow any of them to be installed on your system then
they take precedence over your non-egg packages even you explicitly
put those other packages earlier in your PYTHONPATH. (That latter
behavior is very disagreeable to more than a few prorgammers.)
Sorry if I'm out of the loop and there's some subtlety here that I'm
disregarding, but it doesn't appear that either of the issues you mention is a
actually problem with eggs. These are instead problems with how eggs get
installed by easy_install (which uses a .pth file to extend sys.path). It's
reasonable to put eggs on the PYTHONPATH manually (e.g.
sys.path.append('/path/to/some.egg')) instead of using easy_install to install
them.
I don't think there would be any benefit to changing Python's import machinery
to deal with them; they are essentially just directories (or zipfiles) that
contain packages.
- C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help fixing failing Py3k Unittests in py3k-struni
I have a very remedial question about how to fix test failures due to the side effects of string-unicode integration. The xmlrpc library uses explicit encoding to encode XML tag payloads to (almost always) utf8. Tag literals are not encoded. What would be the best way to mimic this behavior under the new regime? Just use unicode everywhere and encode the entire XML body to utf-8 at the end? Or deal explicitly in bytes everywhere? Or..? Remedially, - C On Jul 10, 2007, at 5:14 PM, Guido van Rossum wrote: One of the most daunting tasks remaining for Python 3.0a1 (to be released by the end of August) is fixing the remaining failing unit tests in the py3k-struni branch (http://svn.python.org/view/python/branches/py3k-struni/). This is the branch where I have started the work on the string/unification branch. I want to promote this branch to become the main Py3k branch ASAP (by renaming it to py3k), but I don't want to do that until all unit tests pass. I've been working diligently on this task, and I've got it down to about 50 tests that are failing on at least one of OSX and Ubuntu (the platforms to which I have easy access). Now I need help. To facilitate distributing the task of getting the remaining tests to pass, I've created a wiki page: http://wiki.python.org/moin/Py3kStrUniTests . Please help! It's easy to help: (1) check out the py3k-struni branch; (2) build it; (3) pick a test and figure out why it's failing; (4) produce a fix; (5) submit the fix to SF (or check it in, if you have submit privileges and are confident enough). In order to avoid duplicate work, I've come up with a simple protocol: you mark a test in the wiki as MINE (with your name) when you start looking at it. You mark it as FIXED [IN SF] once you fix it, adding the patch# if the fix is in SF. If you give up, remove your lock, adding instead a note with what you've found (even just the names of the failing subtests is helpful). Please help! There are other tasks, see PEP 3100. Mail me if you're interested in anything specifically. (Please don't ask me do you think I could do this -- you know better than I whether you're capable of coding at a specific level. If you don't understand the task, you're probably not qualified.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists %40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issues with PEP 3101 (string formatting)
Wrt http://www.python.org/dev/peps/pep-3101/ PEP 3101 says Py3K should allow item and attribute access syntax within string templating expressions but to limit potential security issues, access to underscore prefixed names within attribute/item access expressions will be disallowed. I am a person who has lived with the aftermath of a framework designed to prevent data access by restricting access to underscore- prefixed names (Zope 2, ahem), and I've found it's very hard to explain and justify. As a result, I feel that this is a poor default policy choice for a framework. In some cases, underscore names must become part of an object's external interface. Consider a URL with one or more underscore- prefixed path segment elements (because prefixing a filename with an underscore is a perfectly reasonable thing to do on a filesystem, and path elements are often named after file names) fed to a traversal algorithm that attempts to resolve each path element into an object by calling __getitem__ against the parent found by the last path element's traversal result. Perhaps this is poor design and __getitem__ should not be consulted here, but I doubt that highly because there's nothing particularly special about calling a method named __getitem__ as opposed to some method named traverse. The only precedent within Python 2 for this sort of behavior is limiting access to variables that begin with __ and which do not end with __ to the scope defined by a class and its instances. I personally don't believe this is a very useful feature, but it's still only an advisory policy and you can worm around it with enough gyrations. Given that security is a concern at all, the only truly reasonable way to limit security issues is to disallow item and attribute access completely within the string templating expression syntax. It seems gratuituous to me to encourage string templating expressions with item/attribute access, given that you could do it within the format arguments just as easily in the 99% case, and we've (well... I've) happily been living with that restriction for years now. But if this syntax is preserved, there really should be no *default* restrictions on the traversable names within an expression because this will almost certainly become a hard-to-explain, hard-to-justify bug magnet as it has become in Zope. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Would adding an API for sigprocmask help here? (Although it has been tried before -- http://mail.python.org/ pipermail/python-dev/2003-February/033016.html and died in the womb due to threading-related issues -- http://mail.mems-exchange.org/ durusmail/quixote-users/1248/) - C On Sep 2, 2006, at 8:10 AM, Gustavo Carneiro wrote: We have to resort to timeouts in pygtk in order to catch unix signals in threaded mode. The reason is this. We call gtk_main() (mainloop function) which blocks forever. Suppose there are threads in the program; then any thread can receive a signal (e.g. SIGINT). Python catches the signal, but doesn't do anything; it simply sets a flag in a global structure and calls Py_AddPendingCall(), and I guess it expects someone to call Py_MakePendingCalls(). However, the main thread is blocked calling a C function and has no way of being notified it needs to give control back to python to handle the signal. Hence, we use a 100ms timeout for polling. Unfortunately, timeouts needlessly consume CPU time and drain laptop batteries. According to [1], all python needs to do to avoid this problem is block all signals in all but the main thread; then we can guarantee signal handlers are always called from the main thread, and pygtk doesn't need a timeout. Another alternative would be to add a new API like Py_AddPendingCallNotification, which would let python notify extensions that new pending calls exist and need to be processed. But I would really prefer the first alternative, as it could be fixed within python 2.5; no need to wait for 2.6. Please, let's make Python ready for the enterprise! [2] [1] https://bugzilla.redhat.com/bugzilla/process_bug.cgi#c3 [2] http://perkypants.org/blog/2006/09/02/rfte-python/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists %40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] cgi.FieldStorage DOS (sf bug #1112549)
From the initial bugreport (http://sourceforge.net/tracker/index.php? func=detailaid=1112549group_id=5470atid=105470) Various parts of cgi.FieldStorage call its read_lines_to_outerboundary, read_lines and skip_lines methods. These methods use the readline method of the file object that represents an input stream. The input stream is typically data supplied by an untrusted source (such as a user uploading a file from a web browser). The input data is not required by the RFC 822/1521/1522/1867 specifications to contain any newline characters. For example, it is within the bounds of the specification to supply a a multipart/form-data input stream with a file-data part that consists of a 2GB string composed entirely of x characters (which happens to be something I did that led me to noticing this bug). This bug has been around for about a year but I just worked up a patch yesterday that applies OK against current SVN. It's attached to the issue. Would someone be so kind as to check it in? Guido has already reviewed it, I believe. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
