Re: [Python-Dev] PLY in stdlib (was cffi in stdlib)
On Jul 14, 2013, at 6:32 AM, David Beazley d...@dabeaz.com wrote: I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Doubt about closure/function generation in bytecode
2013/7/12 Andrea Griffini agr...@tin.it: Is there any semantic difference between BUILD_TUPLE 0 LOAD_CONST code object ... MAKE_CLOSURE 0 and LOAD_CONST code object ... MAKE_FUNCTION 0 ? In other words is there any difference between a function and a closure that doesn't capture anything? Is it just a speed optimization for a common case? It probably has more to do with the fact that nested scopes with added much later in Python's history than functions. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 7/15/2013 12:17 AM, Nick Coghlan wrote:
You'd be surprised how many non-core devs react with astonishment
when I suggest that not documenting something isn't enough to avoid
having users consider it a supported public API - they usually get it
after I point out how far you can usually get just by using dir() and
help() to play with a new library at the interactive prompt instead of
looking at out-of-band docs. I figure including the second part will
help prevent some But why? reactions in the future.
I agree with the goal of preventing such reactions, but I suggest
slightly different means to do so.
I just looked and idlelib.__init__.py consists of
# Dummy file to make this a package.
help(idlelib) prints the comment (which surprised me, but it seems to be
smart enough to look for a 'doc comment' when it does not find a doc
string). I now plan to change the comment to a docstring so
'idlelib.__doc__' also works and change the text to something like
A package of private modules that implement the IDLE integrated shell
and editor application. See the manual for the public entry points.
(or maybe list them briefly here in the doc string).
This should properly inform people who introspect.
Also, the manual needs a small, indexed, section on idlelib that says
much the same thing along with documenting what *is* public (which is
not completely clear to me at the moment).
I think Steven has a reasonable point about being clearer that an
explicit leading underscore is the preferred solution for any new
private modules, though, so here's an updated proposal:
The public/private distinction is not always as clean-cut as seems to be
assumed in this discussion. Test and idlelib both have public entry
points to an application based on private files. ('Running the Python
test suite' is a batch application.)
So I do not think either should be _ prefixed. For both, and anything
similar, a proper docstring and manual entry explaining the nuances
should be completely acceptable as a substitute for a blunt,
all-or-nothing flag.
I also think that marking something private, either way, or partially
private with words, should extend to its contents. In particular, the
100s of implementation files in test and idlelib need not be _ marked as
as they would be covered once test and idlelib are properly documented.
The special _ mark makes sense when it is the exception, but not in a
context where it would be the overwhelming norm, because everything but
one or two files is private.
=
Private interfaces
Unless explicitly documented otherwise, a leading underscore on any
name indicates that it is an internal implementation detail and any
backwards compatibility guarantees do not apply. It is strongly
encouraged that private APIs (whether modules, classes, functions,
attributes or other names) be clearly marked in this way, as Python
users may rely on introspection to identify available functionality
and may be misled into believing an API without a leading underscore
is in fact a public API with the standard backwards compatibility
guarantees.
While the explicit use of a leading underscore in the names of private
modules is preferred, all test modules and all modules that are not
explicitly covered in the documentation are also considered private
interfaces, even when their names do not start with a leading
underscore and even if they include a module level documentation
string. This includes submodules of packages that are documented as if
they were a single module.
Since I am planning to explicitly cover idlelib in the doc, to list the
public interface, just as is done for test, I would like it explicitly
mentioned along with test.
I do not think absence of documentation is a good signal. I would rather
say that any private or partially private package that does not have a _
name *must* have a docstring saying that it is all or mostly private.
(Or be a subpackage or module of something so marked.) And presuming
that it is associated with something that is documented, then the doc
should state that the implementation files are private.
In other words, I suggest that modules that are not explicitly covered
in the documentation ... [whose] names do not start with a leading
underscore should be empty. And I think even if they include a module
level documentation string. is backwards. Such files *should* have a
docstring that serves as a substitute for a _ prefix.
Similarly, the specific modules and external APIs imported by a module
are always considered an implementation detail. Other modules should
not rely on indirect access to such imported interfaces unless they
are an explicitly documented part of the containing module's API (such
as ``os.path`` or a package ``__init__`` module that exposes
functionality from submodules).
=
--
Terry Jan Reedy
___
Python-Dev mailing list
Re: [Python-Dev] PLY in stdlib (was cffi in stdlib)
On 15 July 2013 16:01, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Jul 14, 2013, at 6:32 AM, David Beazley d...@dabeaz.com wrote: I honestly don't have any particular thoughts about PLY vs. other parser generators and the merits of their inclusion (or not) in the standard library. I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others. Agreed (and the fact it's a prerequisite for pycparsing and cffi just seals the deal). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PLY in stdlib (was cffi in stdlib)
On 15 July 2013 07:01, Raymond Hettinger raymond.hettin...@gmail.comwrote: I would love to have PLY in the standard library. It would open up a whole new world to some users and be the basis for tool generation for others. +1. Parser generators are useful tools - parsers are right on the boundary of easy enough to understand why you'd like one, but hard enough to put you off implementing our own. And there is value in tools like this being in the stdlib as opposed to a 3rd party dependency. Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding Python scripts to PATHEXT on Windows
With the addition of the Python launcher (PEP 397), Python scripts on Windows are executable in much the same way as on Unix. However, out of the box it is still not quite possible to use Python scripts. This is because the .py extension is not registered with Windows in the PATHEXT environment variable (which holds all file extensions which should be looked for on PATH at the command line). There are two main consequences to this: 1. Under powershell, .py files on PATH are not found at all (this is not the case with cmd.exe, and may be technically a bug in powershell - but it's unlikely to change nevertheless). 2. Under both powershell and cmd, users have to explicitly include the extension when invoking the command. I would like to propose that the .py extension be added to PATHEXT as part of the Register Python Extensions option in the Python MSI installer. This would mean that Python users had a clean way of writing cross-platform wrapper scripts and command-line applications out of the box, just by writing a normal Python script with a #! line at the top. Does anyone have any objections to this? I could try to write a patch, but I know next to nothing about building MSIs, so if any of the installer experts could help that would be fantastic. Paul PS There would still be one difference, in that scripts intended to be used as commands would be named with '.py' on Windows and with no extension on Unix, but that's a pretty trivial difference, and reflects fundamental platform differences rather than configuration. So I don't propose worrying about this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows
Sounds good to me - in the long run it may allow us to drop the Windows exe wrappers when installing scripts on Windows. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows
+1 BTW: I use lots of *.py-executables on Linux and OSX, so different extension don't bother me. Gerald Am 15.07.2013 um 10:26 schrieb Paul Moore p.f.mo...@gmail.com: With the addition of the Python launcher (PEP 397), Python scripts on Windows are executable in much the same way as on Unix. However, out of the box it is still not quite possible to use Python scripts. This is because the .py extension is not registered with Windows in the PATHEXT environment variable (which holds all file extensions which should be looked for on PATH at the command line). There are two main consequences to this: 1. Under powershell, .py files on PATH are not found at all (this is not the case with cmd.exe, and may be technically a bug in powershell - but it's unlikely to change nevertheless). 2. Under both powershell and cmd, users have to explicitly include the extension when invoking the command. I would like to propose that the .py extension be added to PATHEXT as part of the Register Python Extensions option in the Python MSI installer. This would mean that Python users had a clean way of writing cross-platform wrapper scripts and command-line applications out of the box, just by writing a normal Python script with a #! line at the top. Does anyone have any objections to this? I could try to write a patch, but I know next to nothing about building MSIs, so if any of the installer experts could help that would be fantastic. Paul PS There would still be one difference, in that scripts intended to be used as commands would be named with '.py' on Windows and with no extension on Unix, but that's a pretty trivial difference, and reflects fundamental platform differences rather than configuration. So I don't propose worrying about this. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python.00%40klix.ch ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows
On 15 July 2013 09:40, Nick Coghlan ncogh...@gmail.com wrote: Sounds good to me - in the long run it may allow us to drop the Windows exe wrappers when installing scripts on Windows. That's my motivation. In particular removing pip's exe wrappers to ease some of the issues being flagged on distutils-sig. Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Python scripts to PATHEXT on Windows
On 15 July 2013 18:42, Paul Moore p.f.mo...@gmail.com wrote: On 15 July 2013 09:40, Nick Coghlan ncogh...@gmail.com wrote: Sounds good to me - in the long run it may allow us to drop the Windows exe wrappers when installing scripts on Windows. That's my motivation. In particular removing pip's exe wrappers to ease some of the issues being flagged on distutils-sig. Ah, true. For some reason I was thinking we kept __main__.__file__ open, but we don't - we close it as soon as we have read and compiled it. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Refactoring test.support into a subpackage
At the PyCon AU sprints, some of the sprinters worked on a plan Chris Jerdonek and I came up with months ago to convert test.support from a module into a subpackage. This plan arose from some nasty test suite hacks in the lead up to the release of Python 3.3, where some of the pkgutil tests ended up in test.test_runpy so they could use the package creation infrastructure I have in there without needing to extract a helper module with a release imminent. The scope of the plan covers a few distinct steps: 1. Move Lib/test/support.py to Lib/test/support/__init__.py (http://bugs.python.org/issue15494) 2. Add a new (documented) test.support.pkg_helper submodule with the package creation support code (http://bugs.python.org/issue15403 and http://bugs.python.org/issue15376) 3. Move the pkgutil tests to test.test_pkgutil where they belong (http://bugs.python.org/issue15358) 4. Move the existing test.script_helper and test.bytecode_helper support modules into the test.support subpackage and document them (no issue for that as yet) These changes would be made on both 3.3 and default, to avoid future merge conflicts (note that Indra also posted an updated patch for one pending test suite update to cope with the relocation: http://bugs.python.org/issue15415) My main motivation for doing this is to give a bit more visibility to the helpers that already exist, as well as making them easier to find when we want to look into the details of what a test is doing. At the moment, they tend to get lost in the sea of actual test files in the main test directory. (Chris and I initially held off on this while waiting for the final 3.2 release, and then that got delayed and it wasn't until Indra, Michael and Stephen started looking into it at the sprints that I started paying attention again) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hooking into super() attribute resolution
On 9 Jul, 2013, at 1:21, Steve Dower steve.do...@microsoft.com wrote: Except that if it's on a metaclass, the 'instance' it has access to is cls. The descriptor side of things is more interesting, but I see no reason why super can't do that itself, since it knows the actual instance to call __get__ with. (Presumably it already does this with the __dict__ lookup, since that won't call __get__ either.) Explaining the new method is easiest if the default implementation is (literally): def __getlocalname__(self, name): try: return self.__dict__[name] except KeyError: raise AttributeError(name) which does not do any descriptor resolution (and is only a small step from simply replacing __dict__ with a custom object, which is basically where we started). The only change I've really suggested is making it an instance method that can be implemented on a metaclass if you want it for class members. I've documented this (with a different name) in the current PEP draf (http://www.python.org/dev/peps/pep-0447/) and am working on an implementation. Using a lookup method on the metaclass has a nice side-effect: the same method could be used by object.__getattribute__ (aka PyObject_GenericGetAttr) as well which could simplify my primary usecase for this new API. There is a problem with that though: the type attribute cache in Object/typeobject.c, using that cache isn't valid if _PyType_Lookup calls a method instead of peeking in tp_dict (the cache cannot be in the default __getlocalname__ implementation because a primary usecase for the cache appears to be avoiding walking the MRO [1]). I haven't decided yet what to do about this, a number of options: * Don't use __getlocalname__ in _PyType_Lookup * Use __getlocalname__ as a fallback after peeking in tp_dict (that is, use it like __getattr__ instead of __getattribute__) and only cache the attribute when it is fetched from tp_dict. * Don't add a default __getlocalname__ and disable the attribute cache for types with a metatype that does have this method (this might be non-trivial because a metatype might grow a __getlocalname__ slot after instances of the metatype have been created; that would be ugly code but is possbile) Ronald [1] appears because I haven't found documentation for the cache yet ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 447: hooking into super() attribute resolution
Hi,
I've posted a new update of my proposal to add a way to override the attribute
resolution proces by super(). I've rewritten the PEP and implementation based
on feedback by Steve Dower.
In the current edition of the proposal the hook is a method on the type,
defined in a metaclass for types. A simple demo:
class meta (type):
def __locallookup__(cls, name):
return type.__locallookup__(cls, name.upper())
class S (metaclass=meta):
def m(self):
return 42
def M(self):
return fortytwo
class SS (S):
def m(self):
return 21
def M(self):
return twentyone
o = SS()
print(self, o.m())
print(super, super(SS, o).m())
The last line currently prints super 42 and would print super fortytwo with
my proposal.
A major open issue: the __locallookup__ method could also be used for normal
attribute resolution, but that probably causes issues with attribute caching
(see the PEP for details). I haven't worked out yet if it is worthwhile to
tweak the proposal to fix the caching issues (even though the proposal
currently says that PyObject_GenericGetAttr will use the new method). Fixing
the caching issue definitly would help in my primary use case by reducing code
duplication, but might end up making the API unnecessarily complex.
Ronald
PEP: 447
Title: Hooking into super attribute resolution
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren ronaldousso...@mac.com
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, ?
Abstract
In current python releases the attribute resolution of the `super class`_
peeks in the ``__dict__`` attribute of classes on the MRO to look
for attributes. This PEP introduces a hook that classes can use
to override that behavior for specific classes.
Rationale
=
Peeking in the class ``__dict__`` works for regular classes, but can
cause problems when a class dynamicly looks up attributes in a
``__getattribute__`` method.
This new hook makes it possible to affect attribute lookup for both normal
attribute lookup and lookup through the `super class`_.
The superclass attribute lookup hook
Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ in C code) walk an object's MRO and peek in the
class' ``__dict__`` to look up attributes. A way to affect this lookup is
using a method on the meta class for the type, that by default looks up
the name in the class ``__dict__``.
In Python code
--
A meta type can define a method ``__locallookup__`` that is called during
attribute resolution by both ``super.__getattribute__`` and
``object.__getattribute``::
class MetaType (type):
def __locallookup__(cls, name):
try:
return cls.__dict__[name]
except KeyError:
raise AttributeError(name) from None
The example method above is pseudo code for the implementation of this method on
`type`_ (the actual implementation is in C, and doesn't suffer from the
recursion
problem in this example).
The ``__locallookup__`` method has as its arguments a class and the name of the
attribute
that is looked up. It should return the value of the attribute without invoking
descriptors,
or raise `AttributeError`_ when the name cannot be found.
In C code
-
A new slot ``tp_locallookup`` is added to the ``PyTypeObject`` struct, this slot
corresponds to the ``__locallookup__`` method on `type`_.
The slot has the following prototype::
PyObject* (*locallookupfunc)(PyTypeObject* cls, PyObject* name);
This method should lookup *name* in the namespace of *cls*, without looking at
superclasses,
and should not invoke descriptors. The method returns ``NULL`` without setting
an exception
when the *name* cannot be found, and returns a new reference otherwise (not a
borrowed reference).
Usage of this hook
--
The new method will be defined for `type`_, and will peek in the class
dictionary::
static PyObject*
type_locallookup(PyTypeObject* cls, PyObject* name)
{
PyObject* res;
if (!cls-tp_dict) {
return NULL;
}
res = PyDict_GetItem(cls, name);
Py_XINCREF(res);
return res;
}
The new method will be used by both `PyObject_GenericGetAttr`_ and
``super.__getattribute__`` instead of peeking in a type's ``tp_dict``.
Alternative proposals
-
``__getattribute_super__``
..
An earlier version of this PEP used the following static method on classes::
def __getattribute_super__(cls, name, object, owner): pass
This method performed name lookup as well as invoking descriptors and was
necessarily
limited to working only with ``super.__getattribute__``.
Reuse ``tp_getattro``
.
It
Re: [Python-Dev] PEP 447: hooking into super() attribute resolution
On Mon, Jul 15, 2013 at 8:12 AM, Ronald Oussoren ronaldousso...@mac.com wrote: I've posted a new update of my proposal to add a way to override the attribute resolution proces by super(). I've rewritten the PEP and implementation based on feedback by Steve Dower. In the current edition of the proposal the hook is a method on the type, defined in a metaclass for types. A simple demo: class meta (type): def __locallookup__(cls, name): return type.__locallookup__(cls, name.upper()) class S (metaclass=meta): def m(self): return 42 def M(self): return fortytwo class SS (S): def m(self): return 21 def M(self): return twentyone o = SS() print(self, o.m()) print(super, super(SS, o).m()) The last line currently prints super 42 and would print super fortytwo with my proposal. I would suggest adding that motivation to the PEP, since you clearly thought it was necessary to get people to consider your proposal. :-) A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Hm. It looks like the functionality you actually want to hook into is in _PyType_Lookup(). I think that in this case the PEP's acceptance will be highly correlated with your ability to produce an actual patch that (a) implements exactly the functionality you want (never mind whether it matches the exact API currently proposed), and (b) doesn't slow down classes that don't provide this hook. Other than that, I think that it's a laudable attempt at generalizing, and I hope you solve the implementation conundrum. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 447: hooking into super() attribute resolution
On 15 Jul, 2013, at 18:49, Guido van Rossum gu...@python.org wrote: A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Hm. It looks like the functionality you actually want to hook into is in _PyType_Lookup(). That's right. I didn't want to get too technical, but forgot to consider who are reading this :-) I think that in this case the PEP's acceptance will be highly correlated with your ability to produce an actual patch that (a) implements exactly the functionality you want (never mind whether it matches the exact API currently proposed), and (b) doesn't slow down classes that don't provide this hook. I'd only reall need the functional change to super(), but I am working on a patch that also changes _PyType_Lookup. I think I can avoid slowdown by making the tp_locallookup optional and only punishing those classes that use the new slot. A minor complication is that I'll have to change the interface of _PyType_Lookup, it currently returns a borrowed reference and will return a new reference. That's just careful bookkeeping though. Other than that, I think that it's a laudable attempt at generalizing, and I hope you solve the implementation conundrum. I was pleasantly surprised in how the changed API was cleaner and applicable to _PyType_Lookup as well. I guess that means I'm on the right path. Ronald ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #18393: Remove use of deprecated API on OSX
On Mon, Jul 15, 2013 at 11:32 AM, ronald.oussoren python-check...@python.org wrote: http://hg.python.org/cpython/rev/ccbaf6762b54 changeset: 84634:ccbaf6762b54 user:Ronald Oussoren ronaldousso...@mac.com date:Mon Jul 15 18:32:09 2013 +0200 summary: Issue #18393: Remove use of deprecated API on OSX The Gestalt function on OSX is deprecated (starting with OSX 10.8), remove its usage from the stdlib. The patch removes a number of private I believe this means that Lib/test/leakers/test_gestalt.py can be removed as well. functions and a private module, but does not change the public API. The removed code was effectively dead, the platform module has used other code to fetch the OSX version for years and could only use on the Gestalt-based code as a fallback. That fallback can only trigger on broken OSX installs (that is, someone has removed parts of the system install) files: Lib/platform.py| 64 +- Misc/NEWS | 4 + Modules/_gestalt.c | 84 -- setup.py | 4 - 4 files changed, 5 insertions(+), 151 deletions(-) snip diffs -- Repository URL: http://hg.python.org/cpython -- Zach ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 447: hooking into super() attribute resolution
On Mon, Jul 15, 2013 at 9:56 AM, Ronald Oussoren ronaldousso...@mac.com wrote: On 15 Jul, 2013, at 18:49, Guido van Rossum gu...@python.org wrote: A major open issue: the __locallookup__ method could also be used for normal attribute resolution, but that probably causes issues with attribute caching (see the PEP for details). I haven't worked out yet if it is worthwhile to tweak the proposal to fix the caching issues (even though the proposal currently says that PyObject_GenericGetAttr will use the new method). Fixing the caching issue definitly would help in my primary use case by reducing code duplication, but might end up making the API unnecessarily complex. Hm. It looks like the functionality you actually want to hook into is in _PyType_Lookup(). That's right. I didn't want to get too technical, but forgot to consider who are reading this :-) Heh. :-) I think that in this case the PEP's acceptance will be highly correlated with your ability to produce an actual patch that (a) implements exactly the functionality you want (never mind whether it matches the exact API currently proposed), and (b) doesn't slow down classes that don't provide this hook. I'd only reall need the functional change to super(), but I am working on a patch that also changes _PyType_Lookup. I think I can avoid slowdown by making the tp_locallookup optional and only punishing those classes that use the new slot. A minor complication is that I'll have to change the interface of _PyType_Lookup, it currently returns a borrowed reference and will return a new reference. That's just careful bookkeeping though. Hm. That sounds scary. I'd rename it, because this is actually an important (and scary) signature change, that type checkers otherwise won't see. Don't trust that you can find all call sites (there may be some in 3rd party code even though it starts with _). You don't want to suddenly start causing leaks all over the place. FWIW I agree that the change should affect _PyType_Lookup, because it would be very surprising if super() used a different override than regular attribute lookup. Other than that, I think that it's a laudable attempt at generalizing, and I hope you solve the implementation conundrum. I was pleasantly surprised in how the changed API was cleaner and applicable to _PyType_Lookup as well. I guess that means I'm on the right path. There's a Zen of Python line about that. :-) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote: Private interfaces PEP 8 does say: _single_leading_underscore: weak internal use indicator. E.g. from M import * does not import objects whose name starts with an underscore. I'm in favor of making this a stronger recommendation, but I have a small semantic quibble. Instead of private interface it should be non-public interface. The two aren't quite the same thing; for example, often single-leading underscores are used for methods that subclasses are supposed to override, e.g. akin to protected in C++. Besides, Python doesn't really have any strong notion of privateness, so saying non-public means just because you can, doesn't mean you should. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Refactoring test.support into a subpackage
On 7/15/2013 6:22 AM, Nick Coghlan wrote: At the PyCon AU sprints, some of the sprinters worked on a plan Chris Jerdonek and I came up with months ago to convert test.support from a module into a subpackage. This plan arose from some nasty test suite hacks in the lead up to the release of Python 3.3, where some of the pkgutil tests ended up in test.test_runpy so they could use the package creation infrastructure I have in there without needing to extract a helper module with a release imminent. The scope of the plan covers a few distinct steps: 1. Move Lib/test/support.py to Lib/test/support/__init__.py (http://bugs.python.org/issue15494) 2. Add a new (documented) test.support.pkg_helper submodule with the package creation support code (http://bugs.python.org/issue15403 and http://bugs.python.org/issue15376) 3. Move the pkgutil tests to test.test_pkgutil where they belong (http://bugs.python.org/issue15358) 4. Move the existing test.script_helper and test.bytecode_helper support modules into the test.support subpackage and document them (no issue for that as yet) These changes would be made on both 3.3 and default, to avoid future merge conflicts (note that Indra also posted an updated patch for one pending test suite update to cope with the relocation: http://bugs.python.org/issue15415) My main motivation for doing this is to give a bit more visibility to the helpers that already exist, as well as making them easier to find Bundling together all the support code I should know about sounds good to me. I only became aware of script_helper, recently, when I saw it imported in a test_xxx file. I still have no idea what it does. when we want to look into the details of what a test is doing. At the moment, they tend to get lost in the sea of actual test files in the main test directory. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Mon, 15 Jul 2013 15:51:31 -0400 Barry Warsaw ba...@python.org wrote: On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote: Private interfaces PEP 8 does say: _single_leading_underscore: weak internal use indicator. E.g. from M import * does not import objects whose name starts with an underscore. I'm in favor of making this a stronger recommendation, but I have a small semantic quibble. Instead of private interface it should be non-public interface. The two aren't quite the same thing; for example, often single-leading underscores are used for methods that subclasses are supposed to override, e.g. akin to protected in C++. C++ doesn't have a monopoly over the definition of private. Besides, Python doesn't really have any strong notion of privateness, so saying non-public means just because you can, doesn't mean you should. If it doesn't have any notion of privateness, then it can't have a notion of publicness either. If you really want another word (I am personally fine with private), internal it should be. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16 Jul 2013 06:13, Barry Warsaw ba...@python.org wrote: On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote: If you really want another word (I am personally fine with private), internal it should be. I would be fine with internal, since that's how PEP 8 already classifies such names. :) Yeah, internal is a better word for this. Cheers, Nick. -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote: Private interfaces Internal or Non-public Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. How about: All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute. While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring). As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Misc re.match() complaint
In a discussion about mypy I discovered that the Python 3 version of the re module's Match object behaves subtly different from the Python 2 version when the target string (i.e. the haystack, not the needle) is a buffer object. In Python 2, the type of the return value of group() is always either a Unicode string or an 8-bit string, and the type is determined by looking at the target string -- if the target is unicode, group() returns a unicode string, otherwise, group() returns an 8-bit string. In particular, if the target is a buffer object, group() returns an 8-bit string. I think this is the appropriate behavior: otherwise using regular expression matching to extract a small substring from a large target string would unnecessarily keep the large target string alive as long as the substring is alive. But in Python 3, the behavior of group() has changed so that its return type always matches that of the target string. I think this is bad -- apart from the lifetime concern, it means that if your target happens to be a bytearray, the return value isn't even hashable! Does anyone remember whether this was a conscious decision? Is it too late to fix? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum gu...@python.org wrote: In a discussion about mypy I discovered that the Python 3 version of the re module's Match object behaves subtly different from the Python 2 version when the target string (i.e. the haystack, not the needle) is a buffer object. In Python 2, the type of the return value of group() is always either a Unicode string or an 8-bit string, and the type is determined by looking at the target string -- if the target is unicode, group() returns a unicode string, otherwise, group() returns an 8-bit string. In particular, if the target is a buffer object, group() returns an 8-bit string. I think this is the appropriate behavior: otherwise using regular expression matching to extract a small substring from a large target string would unnecessarily keep the large target string alive as long as the substring is alive. But in Python 3, the behavior of group() has changed so that its return type always matches that of the target string. I think this is bad -- apart from the lifetime concern, it means that if your target happens to be a bytearray, the return value isn't even hashable! Does anyone remember whether this was a conscious decision? Is it too late to fix? Hmm, that is not what I'd expect either. I would never expect it to return a bytearray; I'd normally assume that .group() returned a bytes object if the input was binary data and a str object if the input was unicode data (str) regardless of specific types containing the input target data. I'm going to hazard a guess that not much, if anything, would be depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and earlier users are stuck with an extra bytes() call and data copy in these cases I guess. -gps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On 16/07/2013 00:30, Gregory P. Smith wrote:
On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum gu...@python.org
mailto:gu...@python.org wrote:
In a discussion about mypy I discovered that the Python 3 version of
the re module's Match object behaves subtly different from the Python
2 version when the target string (i.e. the haystack, not the needle)
is a buffer object.
In Python 2, the type of the return value of group() is always either
a Unicode string or an 8-bit string, and the type is determined by
looking at the target string -- if the target is unicode, group()
returns a unicode string, otherwise, group() returns an 8-bit string.
In particular, if the target is a buffer object, group() returns an
8-bit string. I think this is the appropriate behavior: otherwise
using regular expression matching to extract a small substring from a
large target string would unnecessarily keep the large target string
alive as long as the substring is alive.
But in Python 3, the behavior of group() has changed so that its
return type always matches that of the target string. I think this is
bad -- apart from the lifetime concern, it means that if your target
happens to be a bytearray, the return value isn't even hashable!
Does anyone remember whether this was a conscious decision? Is it too
late to fix?
Hmm, that is not what I'd expect either. I would never expect it to
return a bytearray; I'd normally assume that .group() returned a bytes
object if the input was binary data and a str object if the input was
unicode data (str) regardless of specific types containing the input
target data.
I'm going to hazard a guess that not much, if anything, would be
depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
earlier users are stuck with an extra bytes() call and data copy in
these cases I guess.
I'm not sure I understand the complaint.
I get this for Python 2.7:
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit
(Intel)] on win
32
Type help, copyright, credits or license for more information.
import array
import re
re.match(ra, array.array(b, a)).group()
array('b', [97])
It's the same even in Python 2.4.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On 16 Jul 2013 09:17, Guido van Rossum gu...@python.org wrote: Does anyone remember whether this was a conscious decision? I doubt it was a conscious decision - an unfortunate amount of the standard library's handling of the text model change falls into the category of implementation accident :( Is it too late to fix? Like Greg, I'm comfortable with the idea of calling bug on this one, fixing it in 3.4 and making a note in the Porting to Python 3.4 section of the What's New guide. Cheers, Nick. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On Mon, Jul 15, 2013 at 5:10 PM, MRAB pyt...@mrabarnett.plus.com wrote:
On 16/07/2013 00:30, Gregory P. Smith wrote:
On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum gu...@python.org
mailto:gu...@python.org wrote:
In a discussion about mypy I discovered that the Python 3 version of
the re module's Match object behaves subtly different from the Python
2 version when the target string (i.e. the haystack, not the needle)
is a buffer object.
In Python 2, the type of the return value of group() is always either
a Unicode string or an 8-bit string, and the type is determined by
looking at the target string -- if the target is unicode, group()
returns a unicode string, otherwise, group() returns an 8-bit string.
In particular, if the target is a buffer object, group() returns an
8-bit string. I think this is the appropriate behavior: otherwise
using regular expression matching to extract a small substring from a
large target string would unnecessarily keep the large target string
alive as long as the substring is alive.
But in Python 3, the behavior of group() has changed so that its
return type always matches that of the target string. I think this is
bad -- apart from the lifetime concern, it means that if your target
happens to be a bytearray, the return value isn't even hashable!
Does anyone remember whether this was a conscious decision? Is it too
late to fix?
Hmm, that is not what I'd expect either. I would never expect it to
return a bytearray; I'd normally assume that .group() returned a bytes
object if the input was binary data and a str object if the input was
unicode data (str) regardless of specific types containing the input
target data.
I'm going to hazard a guess that not much, if anything, would be
depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
earlier users are stuck with an extra bytes() call and data copy in
these cases I guess.
I'm not sure I understand the complaint.
I get this for Python 2.7:
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on
win
32
Type help, copyright, credits or license for more information.
import array
import re
re.match(ra, array.array(b, a)).group()
array('b', [97])
It's the same even in Python 2.4.
Ah, but now try it with buffer():
re.search('yz+', buffer('xyzzy')).group()
'yzz'
The equivalent in Python 3 (using memoryview) returns a memoryview:
re.search(b'yz+', memoryview(b'xyzzy')).group()
memory at 0x10d03a688
And I still think that any return type for group() except bytes or str
is wrong. (Except possibly a subclass of these.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Is it safe to call PyEval_EvalFrameEx() with an exception set?
Hi,
I'm working on the issue #18408 (fix issues found by my pyfailmalloc
tool). To analyze some bugs, I have to add debug traces in various
functions to find which function returned NULL without setting an
error, or the opposite: returned a valid object, but with an exception
set.
I would like to add assertions in Python/ceval.c to detect such bugs
earlier. The problem is that some functions rely on the ability to
call PyEval_EvalFrameEx() with an exception set. Is it expected?
Should it be avoided? The current exception can be replaced with a new
exception.
Example 1:
module = PyImport_Import(module_name);
if (module == NULL) {
PyErr_Format(PicklingError,
Can't pickle %R: import of module %R failed,
obj, module_name);
goto error;
}
Formatting the obj argument calls indirectly PyEval_EvalFrameEx(),
whereas an ImportError was raised. The original ImportError is
replaced with a new PickleError exception.
To not call PyEval_EvalFrameEx() with an exception set, PyErr_Format()
can call PyErr_Clear() before calling PyUnicode_FromFormatV(), maybe
only in debug mode.
Example 2:
test_sqlite uses sqlite.connection.create_aggregate() to create a
Python callback called by the C sqlite3 library. The callback is a
class with 3 methods: constructor, step() and finalize(). The test
AggregateTests.CheckAggrNoStep() uses a class with no step() method:
it ensures that an AttributeError is raised.
The problem is that the finalize() method is called even if the class
has no step method, and so there is a current AttributeError
exception.
I don't know if finalize() should be called in this case. It may be a
bug in the sqlite module. More generally,
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the non-internal API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to PEP8-ify my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. - C On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote: Private interfaces Internal or Non-public Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. How about: All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute. While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring). As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16/07/13 10:23, Chris McDonough wrote: If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to PEP8-ify my code in some limited-value-added way) will be harmed. I sympathise with your pain, but is that not true of every PEP 8 naming convention? They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. I have better things to do than PEP8-ify old, working, stable code is a perfectly acceptable answer. I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so might be an even better one :-) -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16 July 2013 10:23, Chris McDonough chr...@plope.com wrote: On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the non-internal API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to PEP8-ify my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. Hence the just say it's an internal API in the docstring escape clause. That's a pretty low bar to meet for PEP 8 compliance, and one we need, too (for things like test, idlelib and a bundled pip). I do need to figure out how to tweak the wording to make it clear that internal nature of an API applies to all contained APIs as well. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
Guido van Rossum writes: And I still think that any return type for group() except bytes or str is wrong. (Except possibly a subclass of these.) I'm not sure I understand. Do you mean in the context of the match object API, where constructing (target, match.start(), match.end()) to get a group-like object that refers to the target rather than copying the text is simple? (Such objects are very useful in the restricted application of constructing a programmable text editor.) Or is this something deeper, that a group *is* a new object in principle? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On 16/07/2013 01:25, Guido van Rossum wrote:
On Mon, Jul 15, 2013 at 5:10 PM, MRAB pyt...@mrabarnett.plus.com wrote:
On 16/07/2013 00:30, Gregory P. Smith wrote:
On Mon, Jul 15, 2013 at 4:14 PM, Guido van Rossum gu...@python.org
mailto:gu...@python.org wrote:
In a discussion about mypy I discovered that the Python 3 version of
the re module's Match object behaves subtly different from the Python
2 version when the target string (i.e. the haystack, not the needle)
is a buffer object.
In Python 2, the type of the return value of group() is always either
a Unicode string or an 8-bit string, and the type is determined by
looking at the target string -- if the target is unicode, group()
returns a unicode string, otherwise, group() returns an 8-bit string.
In particular, if the target is a buffer object, group() returns an
8-bit string. I think this is the appropriate behavior: otherwise
using regular expression matching to extract a small substring from a
large target string would unnecessarily keep the large target string
alive as long as the substring is alive.
But in Python 3, the behavior of group() has changed so that its
return type always matches that of the target string. I think this is
bad -- apart from the lifetime concern, it means that if your target
happens to be a bytearray, the return value isn't even hashable!
Does anyone remember whether this was a conscious decision? Is it too
late to fix?
Hmm, that is not what I'd expect either. I would never expect it to
return a bytearray; I'd normally assume that .group() returned a bytes
object if the input was binary data and a str object if the input was
unicode data (str) regardless of specific types containing the input
target data.
I'm going to hazard a guess that not much, if anything, would be
depending on getting a bytearray out of that. Fix this in 3.4? 3.3 and
earlier users are stuck with an extra bytes() call and data copy in
these cases I guess.
I'm not sure I understand the complaint.
I get this for Python 2.7:
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on
win
32
Type help, copyright, credits or license for more information.
import array
import re
re.match(ra, array.array(b, a)).group()
array('b', [97])
It's the same even in Python 2.4.
Ah, but now try it with buffer():
re.search('yz+', buffer('xyzzy')).group()
'yzz'
The equivalent in Python 3 (using memoryview) returns a memoryview:
re.search(b'yz+', memoryview(b'xyzzy')).group()
memory at 0x10d03a688
And I still think that any return type for group() except bytes or str
is wrong. (Except possibly a subclass of these.)
On the other hand, I think that it's not unreasonable that the output
is the same type as the input. You could reason that what it's doing is
returning a slice of the input, and that slice should be the same type
as its source.
Incidentally, the regex module does what Python 3's re module currently
does, even in Python 2. Nobody's complained!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote: If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. I have better things to do than PEP8-ify old, working, stable code is a perfectly acceptable answer. I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so might be an even better one :-) Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code. /scarcasm - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On Mon, Jul 15, 2013 at 7:18 PM, MRAB pyt...@mrabarnett.plus.com wrote: On the other hand, I think that it's not unreasonable that the output is the same type as the input. You could reason that what it's doing is returning a slice of the input, and that slice should be the same type as its source. By now I'm pretty sure that is why it changed. But I am challenging how useful that is, compared to always returning something immutable. Incidentally, the regex module does what Python 3's re module currently does, even in Python 2. Nobody's complained! Well, you'd only see complaints from folks who (a) use the regex module, (b) use it with a buffer object as the target string, and (c) try to use the group() return value as a dict key. Each of these is probably a small majority of all users. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16 July 2013 12:20, Chris McDonough chr...@plope.com wrote: On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote: If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. I have better things to do than PEP8-ify old, working, stable code is a perfectly acceptable answer. I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so might be an even better one :-) Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code. /scarcasm How do get from If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring to updating 500k lines of public code? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. Cheers, Nick. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (merge default - 3.3): merge 3.3 (#18470)
2013/7/16 benjamin.peterson python-check...@python.org: http://hg.python.org/cpython/rev/8889c9b5dd3a changeset: 84654:8889c9b5dd3a branch: 3.3 parent: 84653:c3a510b22218 parent: 84652:8a078bf3cf14 user:Benjamin Peterson benja...@python.org date:Mon Jul 15 19:15:49 2013 -0700 summary: merge 3.3 (#18470) This commit merges branches default and 3.3 into 3.3. Usually, we merge branches 3.3 and default into default. Because of this changeset, when I try to merge 3.3 into default, I get a long list of modifications... Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470)
2013/7/16 benjamin.peterson python-check...@python.org:
http://hg.python.org/cpython/rev/c3a510b22218
changeset: 84653:c3a510b22218
branch: 3.3
parent: 84651:e22dd5fda5a8
user:Benjamin Peterson benja...@python.org
date:Mon Jul 15 19:15:34 2013 -0700
summary:
check the return value of new_string() (closes #18470)
...
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -250,17 +253,21 @@
t++;
if (begin t) {
-char* r = new_string(begin, t - begin);
+char* r = new_string(begin, t - begin, tok);
+if (!r)
+return 0;
char* q = get_normal_name(r);
...
Visual Studio does support instructions between declarations, and so
this changeset broke Windows buildbots.
Should we add the -Werror=declaration-after-statement compiler flag
to the 3.3 branch? (in debug mode?)
Victor
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Jul 15, 2013, at 08:23 PM, Chris McDonough wrote: On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote: Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the non-internal API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to PEP8-ify my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. I can sympathize with this. What I tried to get at, but probably didn't succeed, is to say if it starts with a single underscore, treat it as internal but not necessarily the converse, i.o.w. if it *doesn't* start with an underscore that does not imply that it's public. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 12:34 +1000, Nick Coghlan wrote: How do get from If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring to updating 500k lines of public code? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. I may be wrong, but as I see it this is a decision to help (although only conditionally!) a theoretical IDE user or an agressive user of dir() at the expense of effectively either 1) making existing maintainers revisit existing code, or 2 ) making maintainers continually explain to users that their codebase is not PEP8 compliant. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. Fair enough, technically. But historical reality is that a lot of people just haven't read much past the title: Style Guide for Python Code. Or at least they appear to have no concept that it's about code *in the stdlib only*. People take it way, way too seriously for non-stdlib code. But they do, and it takes time to manage that. Given that misunderstanding, is there a way to divorce stdlib-centric guidelines like the one being discussed now from PEP8-ness? I don't think any amount of marketing effort or reasoned explanation is going to separate PEP8 from correct code in people's minds at this point. - C ps.. the real irritant behind my minirant is this: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Misc re.match() complaint
On 16 July 2013 12:20, Guido van Rossum gu...@python.org wrote: On Mon, Jul 15, 2013 at 7:03 PM, Stephen J. Turnbull step...@xemacs.org wrote: Or is this something deeper, that a group *is* a new object in principle? No, I just think of it as returning a string and I think it's most useful if that is always an immutable object, even if the target string is some other bytes buffer. FWIW, it feels as if the change in behavior is probably just due to how slices work. I took a look at the way the 2.7 re code works, and the change does indeed appear to be due to the difference in the way slices work for buffer and memoryview objects: Slicing a buffer creates an 8-bit string: buffer(babc)[0:1] 'a' Slicing a memoryview creates another memoryview: memoryview(babc)[0:1] memory at 0x7f3320541b98 Unfortunately, memoryview doesn't currently allow subclasses, so it isn't easy to create a derivative that coerces to bytes on slicing :( Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16 July 2013 13:02, Chris McDonough chr...@plope.com wrote: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as all Python code in the world should follow these rules cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On Tue, 2013-07-16 at 13:11 +1000, Nick Coghlan wrote: On 16 July 2013 13:02, Chris McDonough chr...@plope.com wrote: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user. You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as all Python code in the world should follow these rules cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have PEP8 modes turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. - C ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.3): check the return value of new_string() (closes #18470)
2013/7/15 Victor Stinner victor.stin...@gmail.com:
2013/7/16 benjamin.peterson python-check...@python.org:
http://hg.python.org/cpython/rev/c3a510b22218
changeset: 84653:c3a510b22218
branch: 3.3
parent: 84651:e22dd5fda5a8
user:Benjamin Peterson benja...@python.org
date:Mon Jul 15 19:15:34 2013 -0700
summary:
check the return value of new_string() (closes #18470)
...
diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c
--- a/Parser/tokenizer.c
+++ b/Parser/tokenizer.c
@@ -250,17 +253,21 @@
t++;
if (begin t) {
-char* r = new_string(begin, t - begin);
+char* r = new_string(begin, t - begin, tok);
+if (!r)
+return 0;
char* q = get_normal_name(r);
...
Visual Studio does support instructions between declarations, and so
this changeset broke Windows buildbots.
Fixed.
Should we add the -Werror=declaration-after-statement compiler flag
to the 3.3 branch? (in debug mode?)
We've lived with this limitation for years. I'm sure we can wait until
3.4 becomes maintenance. :)
--
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
On 16 July 2013 13:16, Chris McDonough chr...@plope.com wrote: I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have PEP8 modes turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-fly-improved-doctest-support-and-more-in-pycharm-2-7/ It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as import pip rather than import _pip and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up. Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013. However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
