Re: [Python-Dev] PEP 488: elimination of PYO files
On 2015-03-06 11:34 AM, Brett Cannon wrote: > This PEP proposes eliminating the concept of PYO files from Python. > To continue the support of the separation of bytecode files based on > their optimization level, this PEP proposes extending the PYC file > name to include the optimization level in bytecode repository > directory (i.e., the ``__pycache__`` directory). As a packager, this PEP is a bit silent on it's expectations about what will happen with (for instance) Debian and Fedora packages for Python. My familiarity is with Fedora, and on that platform, we ship .pyc and .pyo files (using -O for the .pyo). Is it your expectation that such platforms will still distribute -O only? Or also -OO? In my world, all of the __pycache__ directories are owned by root. -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask
On 2015-03-12 10:46 AM, Eric V. Smith wrote:
> On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
>> Related issue:
>>
>>>>> ipaddress.IPv4Interface('10.0.0.255/8') + 1
>> IPv4Interface('10.0.1.0/32')
>>
>> Should the result be IPv4Interface('10.0.0.0/8')
>> (i.e. wrap around)? Should OverflowError be raised?
>
> I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
> 10.0.0.0/8 is unlikely to be desirable.
>
No, you are both imbuing meaning to the dot-notation that is not there.
A.B.C.D is just an ugly way to represent a single 32-bit unsigned
integer. The only contentious part would be when your addition would
overlfow the host-portion of the address:
>>> ipaddress.IPv4Interface('10.255.255.255/8') + 1
IPv4Interface('11.0.0.0/8')
However, if you take the perspective that the address is just a 32-bit
unsigned integer, then it makes perfect sense. I would argue it's likely
to be a source of bugs, but I can't say either way because I never
adopted using this library due to issues that are cataloged in the
mailing list archives. (In the end, I wrote my own and have never found
the need to support such operands.)
--
Scott Dial
[email protected]
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.x vs 3.x survey results
On 2014-01-02 17:54, Dan Stromberg wrote: > I put it at https://wiki.python.org/moin/2.x-vs-3.x-survey It would've been nice to see some crosstabs. Pretty much any question after Q3 is incomprehensible without splitting the respondents into sub-groups first. Of the 2.49% of people who said they've never written Python 2.x, how many of those people also said they have never written Python 3.x too? (There really is 4 categories of developers being surveyed here.) Of the 22.91% of people who said Python 3.x was a mistake, how many of them also said they have never written any Python 3.x? Of the 40% of people who said they have never written Python 3.x, how many of them also said they had dependencies keeping them on Python 2.x? Etc. -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake
On 2014-01-11 22:09, Nick Coghlan wrote:
> For Python 2 folks trying to grok where the "bright line" is in terms of
> the Python 3 text model: if your proposal includes *any* kind of
> implicit serialisation of non binary data to binary, it is going to be
> rejected as an addition to the core bytes type. If it avoids crossing
> that line (as the buffer-API-only version of PEP 460 does), then we can
> talk.
To take such a hard-line stance, I would expect you to author a PEP to
strip the ASCII conveniences from the bytes and bytearray types.
Otherwise, I find it a bit schizophrenic to argue that methods like
lower, upper, title, and etc. don't implicitly assume encoding:
>>> a = "scott".encode('utf-16')
>>> b = a.title()
>>> c = b.decode('utf-16')
'SCOTT'
So, clearly title() not only depends on the bytes characters encoded in
a superset of ASCII characters, it depends on the bytes being a sequence
of ASCII characters, which looks an awful lot like an operation on an
implicit encoded string.
>>> b"文字化け"
File "", line 1
SyntaxError: bytes can only contain ASCII literal characters.
There is an implicit serialization right there. My terminal is utf8 (or
even if my source encoding is utf8), so why would that not be:
b'\xe6\x96\x87\xe5\xad\x97\xe5\x8c\x96\xe3\x81\x91'
I sympathize with Ethan that the bytes and bytearray types already seem
to concede that bytes is the type you want to use for 7-bit ASCII
manipulations. If that is not what we want, then we are not doing a good
job communicating that to developers with the API. At the onset, the
bytes literal itself seems to be an attractive nuisance as it gives a
nod to using bytes for ASCII character sequences (a.k.a ASCII strings).
Regards,
-Scott
--
Scott Dial
[email protected]
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 2014-01-22 9:33 AM, Donald Stufft wrote: > For everything but pip, you’d add it to your OS cert store. Pip doesn’t > use that so you’d have to use the —cert config. What if I don't want that self-signed cert to be trusted by all users on the system? What if I don't have administrative rights? How do I do it then? Is this common knowledge for average users? Are we trading one big red box in the documentation for another? Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/12/2013 10:51 PM, Steven D'Aprano wrote: > And two examples from asm-generic/errno.h: > > #define EWOULDBLOCK EAGAIN /* Operation would block */ > #define EDEADLOCK EDEADLK > That's actually even better of an example than you may have realized because historically EWOULDBLOCK != EAGAIN[1]. So, there very well may need to exist such code as: if : _EAGAIN = _EWOULDBLOCK = else: _EAGAIN = _EWOULDBLOCK = class Errno(Enum): EAGAIN = _EAGAIN EWOULDBLOCK = _EWOULDBLOCK I don't think it's all that uncommon that enum values that represent states of a system get merged or renamed over time, and this one is a great example of that. [1] http://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Error-Codes.html#index-EAGAIN-97 -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/23/2013 10:53 AM, R. David Murray wrote: > Ah. I'd be looking for a bug every time I saw isinstance(value, > myEnumClass). A better class name for values and an API for > getting that class from the EnumClass would be nice, though. > (So that you could write "isinstance(value, MyEnumClass.ValueClass)", > say.) Reading what you have wrote, it seems that the issue is whether you consider an instance of Enum a "thing" or a "class of things". If you think of it as a "thing", then "C" is a object that has attributes for other "things" that are not like "C". However, if you think of "C" as a "class of things", then "C" having attributes that are instances of it's type is completely natural. Fundamentally, the question is whether an instance of Enum is a new type or an instance. And for me, it's a new type and I expect enum values to be instance of that type. -Scott -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/23/2013 11:58 AM, Guido van Rossum wrote: > You seem to be mixing up classes and metaclasses. I was trying to explain it in more plain terms, but maybe I made it even more confusing.. Anyways, I agree with you that isinstance(Color.RED, Color) should be True. -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 6/18/2013 4:40 PM, Victor Stinner wrote:
> No context argument
> ---
>
> Simplify the signature of allocator functions, remove the context
> argument:
>
> * ``void* malloc(size_t size)``
> * ``void* realloc(void *ptr, size_t new_size)``
> * ``void free(void *ptr)``
>
> It is likely for an allocator hook to be reused for
> ``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
> ``PyMem_SetRawAllocator()``, but the hook must call a different function
> depending on the allocator. The context is a convenient way to reuse the
> same custom allocator or hook for different Python allocators.
I think there is a lack of justification for the extra argument, and the
extra argument is not free. The typical use-case for doing this
continuation-passing style is when the set of contexts is either
unknown, arbitrarily large, or infinite. In other words, when it would
be either impossible or impractical to enumerate all of the contexts.
However, in this case, we have only 3.
Your proposal already puts forward having 3 pairs of Get/Set functions,
so there is no distinct advantage in having a single typedef instance
that you pass in to all 3 of them. And, having all 3 pairs use the same
typedef is a bit of an attractive nuisance, in that one could pass the
wrong allocators to the wrong setter. With that, I could argue that
there should be 3 typedefs to prevent coding errors.
Nevertheless, the ctx argument buys the implementer nothing if they have
to begin their alloc function with "if(ctx == X)". In other words, there
is nothing simpler about:
"""
void *_alloc(void *ctx, size_t size) {
if(ctx == PYALLOC_PYMEM)
return _alloc_pymem(size);
else if(ctx == PYALLOC_PYMEM_RAW)
return _alloc_pymem_raw(size);
else if(ctx == PYALLOC_PYOBJECT)
return _alloc_pyobject(size);
else
abort();
}
PyMemBlockAllocator pymem_allocator =
{.ctx=PYALLOC_PYMEM, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pymem_raw_allocator =
{.ctx=PYALLOC_PYMEM_RAW, .alloc=&_alloc, .free=&_free};
PyMemBlockAllocator pyobject_allocator =
{.ctx=PYALLOC_PYOBJECT, .alloc=&_alloc, .free=&_free};
"""
In comparison to:
"""
PyMemBlockAllocator pymem_allocator =
{.alloc=&_alloc_pymem, .free=&_free_pymem};
PyMemBlockAllocator pymem_raw_allocator =
{.alloc=&_alloc_pymem_raw, .free=&_free_pymem};
PyMemBlockAllocator pyobject_allocator =
{.alloc=&_alloc_pyobject, .free=&_free_pyobject};
"""
And in the latter case, there is no extra indirect branching in the
hot-path of the allocators.
Also, none of the external libraries cited introduce this CPS/ctx stuff.
--
Scott Dial
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 6/18/2013 11:32 PM, Nick Coghlan wrote: > Agreed more of that rationale needs to be moved from the issue tracker > into the PEP, though. Thanks for the clarification. I hadn't read the issue tracker at all. On it's face value, I didn't see what purpose it served, but having read Kristján's comments on the issue tracker, he would like to store state for the allocators in that ctx pointer.[1] Having read that (previously, I thought the only utility was distinguishing which domain it was -- a small, glorified enumeration), but his use-case makes sense and definitely is informative to have in the PEP, because the utility of that wasn't obvious to me. Thanks, -Scott [1] http://bugs.python.org/issue3329#msg190529 """ One particular trick we have been using, which might be of interest, is to be able to tag each allocation with a "context" id. This is then set according to a global sys.memcontext variable, which the program will modify according to what it is doing. This can then be used to track memory usage by different parts of the program. """ -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
> Many people have raised concerns about this change, so I've now backed it out.
I think that change also goes with this change:
http://hg.python.org/cpython/rev/f1dc30a1be72
changeset 84248:f1dc30a1be72 2.7
Arrange structure to match the common access patterns.
1.1 --- a/Modules/_collectionsmodule.c
1.2 +++ b/Modules/_collectionsmodule.c
1.3 @@ -48,8 +48,8 @@
1.4
1.5 typedef struct BLOCK {
1.6 struct BLOCK *leftlink;
1.7 +PyObject *data[BLOCKLEN];
1.8 struct BLOCK *rightlink;
1.9 -PyObject *data[BLOCKLEN];
1.10 } block;
1.11
1.12 #define MAXFREEBLOCKS 10
, which seems like a strange micro-optimization, at best.
Based on that structure, it would seem that neither BLOCKLEN being 62
(previous value) nor 64 (the new value) make much sense. It would seem
best that sizeof(block) == 64, so BLOCKLEN should be (64 -
2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
this structure provides any meaningful performance improvement.
--
Scott Dial
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 6/23/2013 8:16 PM, Raymond Hettinger wrote: > Yes, this is a micro-optimization. In working on implementing > deque slicing for 3.4, I restudied the block access patterns. > On an appendleft(), popleft() or extendleft() operation, the left link is > accessed immediately before or after the leftmost entry in the data block. > The old struct arrangement can cause an unnecessary cache miss > when jumping leftward. This was something I overlooked when I > originally wrote the code almost a decade ago. A decade ago, cache lines were 64 bytes, pointers were 4 bytes, and allocations were 16 byte aligned, so there could never be a cache miss. Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and we still allocating on 16 byte alignment, so you have a 25% chance of a cache miss now. -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
On 8/26/2013 8:51 AM, Antoine Pitrou wrote: > Le Mon, 26 Aug 2013 08:24:58 -0400, > Tres Seaver a écrit : >> On 08/26/2013 04:36 AM, Antoine Pitrou wrote: >>> event-driven processing using network libraries >> >> Maybe I missed something: why should considerations from that topic >> influence the design of an API for XML processing? > > Because this API is mostly useful when the data is received (*) at a > slow enough speed - which usually means from the network, not from a > hard drive. ... > The whole *point* of adding IncrementalParser was to parse incoming > XML data in a way that is friendly with event-driven network > programming, other use cases being *already* covered by existing > APIs. This is why it's far from nonsensical to re-use an existing > terminology from that world. Since when is Tulip the OOWTDI? If this was Twisted, it would be "write" and "finish"[1]. Tulip's Protocol ABC isn't even a good match for the application. There is reason that Twisted has a separate Consumer/Producer interface from the network I/O interface. I'm sure there is other existing practice in this specific area too (e.g., XMLParser). [1] http://twistedmatrix.com/documents/13.1.0/api/twisted.protocols.ftp.IFinishableConsumer.html -- Scott Dial [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Aahz wrote: > On Wed, Mar 11, 2009, Antoine Pitrou wrote: >> After Hrvoje's message, let me rephrase my suggestion. Let's instead allow: >>open(..., sync_on="close") >>open(..., sync_on="flush") >> >> with a default of None meaning no implicit syncs. > > That looks good, though I'd prefer using named constants rather than > strings. I would agree, but where do you put them? Since open is a built-in, where would you suggest placing such constants (assuming we don't want to pollute the built-in namespace)? -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing on Solaris
Jesse Noller wrote: > On Fri, Mar 20, 2009 at 8:50 PM, Christian Heimes wrote: >> Martin v. Löwis schrieb: >>>> Today I was in contact with a Python user who tried to compile >>>> pyprocessing - the ancestor of multiprocessing - on Solaris. It failed >>>> to run because Solaris is missing two features (HAVE_FD_TRANSFER and >>>> HAVE_SEM_TIMEDWAIT). Does anybody have a Solaris box at his disposal to >>>> test the settings? Neither Python 2.6 nor my backup have the correct >>>> settings for Solaris. >>> I don't quite understand what it is that you want tested - what >>> "settings"? >>> >>> Most likely, the answer is yes, I can test stuff on Solaris (both SPARC >>> and x86/amd64). >> According to the user's experience multiprocessing should not compile >> and run correctly unless this patch is applied. I'm not sure if the >> value "solaris" for platform is correct. You may also need to change >> libraries to ['rt']. >> >> >> Index: setup.py >> === >> --- setup.py(revision 70478) >> +++ setup.py(working copy) >> @@ -1280,6 +1280,14 @@ >> ) >> libraries = [] >> >> +elif platform == 'solaris': >> +macros = dict( >> +HAVE_SEM_OPEN=1, >> +HAVE_SEM_TIMEDWAIT=0, >> +HAVE_FD_TRANSFER=0, >> +) >> +libraries = [] >> + >> else: # Linux and other unices >> macros = dict( >> HAVE_SEM_OPEN=1, > > If this should be addressed in trunk/3k, we need to track this in the > tracker in the bug I cited in the other email. I can't speak for the > original pyprocessing code. > I just checked out the trunk on a Sparc Solaris 8 box, and on the trunk, those defines are specified differently: building '_multiprocessing' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=1 -IModules/_multiprocessing -I. -I./Include -I/usr/local/include -IInclude -I/nfs/nfs2/home/scratch/scodial/python-trunk -c trunk/Modules/_multiprocessing/multiprocessing.c -o build/temp.solaris-2.8-sun4u-2.7/trunk/Modules/_multiprocessing/multiprocessing.o However, the build is still without issue: trunk/Modules/_multiprocessing/multiprocessing.c: In function `multiprocessing_sendfd': trunk/Modules/_multiprocessing/multiprocessing.c:100: warning: implicit declaration of function `CMSG_SPACE' trunk/Modules/_multiprocessing/multiprocessing.c:117: warning: implicit declaration of function `CMSG_LEN' trunk/Modules/_multiprocessing/connection.h: In function `connection_new': trunk/Modules/_multiprocessing/connection.h:51: warning: unknown conversion type character `z' in format trunk/Modules/_multiprocessing/connection.h:51: warning: too many arguments for format trunk/Modules/_multiprocessing/connection.h: In function `connection_repr': trunk/Modules/_multiprocessing/connection.h:401: warning: unknown conversion type character `z' in format trunk/Modules/_multiprocessing/connection.h: In function `connection_new': trunk/Modules/_multiprocessing/connection.h:51: warning: unknown conversion type character `z' in format trunk/Modules/_multiprocessing/connection.h:51: warning: too many arguments for format trunk/Modules/_multiprocessing/connection.h: In function `connection_repr': trunk/Modules/_multiprocessing/connection.h:401: warning: unknown conversion type character `z' in format -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing on Solaris
Christian Heimes wrote: > Martin v. Löwis schrieb: >>> According to the user's experience multiprocessing should not compile >>> and run correctly unless this patch is applied. >> Can this please be more qualified? I can confirm Scott's observation: >> for the trunk, it compiles just fine, using SunPro CC on Solaris 10, >> on SPARC. Also, test_multiprocessing passes. Sorry, I mistakenly said "without issue" and then copied the issues below. I meant to say "not without issues." _multiprocessing does *not* build on Solaris 8. >> IIUC, incorrect setting of HAVE_SEM_TIMEDWAIT to 1 should cause a >> compile failure, as sem_timedwait would be called but not defined. >> However, sem_timedwait *is* defined on Solaris. >> >> Likewise, for HAVE_FD_TRANSFER=1 to work, SCM_RIGHTS must be defined >> (and implemented); it is defined in sys/socket.h, and documented in >> socket.h(3HEAD). >> >> So there must be going on something else at the user's machine. > > The user doesn't respond to my inquiries anymore. According to his > initial message he is using Solaris 5.8 with GCC 3.4.6 on a Sun Fire > machine. Here is a link to his mesage: > http://permalink.gmane.org/gmane.comp.python.general/615802 I can confirm his build issues on: $ uname -srvmpi SunOS 5.8 Generic_117350-51 sun4u sparc SUNW,Sun-Fire-280R $ gcc -v Reading specs from /usr/local/gnu/lib/gcc-lib/sparc-sun-solaris2.8/2.95.2/specs gcc version 2.95.2 19991024 (release) My build output from within the trunk is the same as his modulo the details of being part of a trunk build instead. Is Solaris 8 really a supported platform? If so, I can investigate the changes he suggested. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Multiprocessing on Solaris
Martin v. Löwis wrote: >> Sorry, I mistakenly said "without issue" and then copied the issues >> below. I meant to say "not without issues." _multiprocessing does *not* >> build on Solaris 8. > > Hmm. They are all warnings - did you omit the actual error message? > > The lack of CMSG_LEN seems to suggest that control messages are not > supported on Solaris 8, or that you need to include an additional > header file to get them. Sorry, I was trying to keep the amount of noise to a minimum, but those particular implicit function declarations ultimately lead to linking errors for lack of those same symbols: *** WARNING: renaming "_multiprocessing" since importing it failed: ld.so.1: python: fatal: relocation error: file build/lib.solaris-2.8-sun4u-2.7/_multiprocessing.so: symbol CMSG_SPACE: referenced symbol not found >> Is Solaris 8 really a supported platform? If so, I can investigate the >> changes he suggested. > > The concept of "supported platform" doesn't really exist - there is no > way to obtain "support". If it works, it works, if it doesn't, it > doesn't. So if you want it fixed, provide a patch - else we can drop > the issue. I have no personal interest in the matter. I just happened to have access to an older Solaris just for this sort of testing. If someone has a patch, then I would be glad to test it, but otherwise, I am not going to invest any time on this. The patch given by the OP is clearly undesirable since status quo works just fine for a modern Solaris install. > I think multiprocessing is misguided in hard-coding these settings > into setup.py. As we can see, the necessary features are available > on some versions of Solaris, but not on others. It would be better > if autoconf tests were written, and the entire configuration removed > from setup.py. I agree, but I am not familiar enough with the set of features that multiprocessing is implicitly depending on and how to test for them on all of the platforms. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 380 (yield from a subgenerator) comments
P.J. Eby wrote: > One remaining quirk or missing piece: ISTM there needs to be a way to > extract the return value without using a yield-from statement. I mean, > you could write a utility function like: > >def unyield(geniter): >try: >while 1: geniter.next() >except GeneratorReturn as v: >return v.value My first thought was to ask why it was not equivalent to say: x = yield g x = yield from g This would seem like a more obvious lack of parallelism to pick on wrt. return values. This unyield() operation seems contrived. Never before have you been able to write a generator that returns a value, why would these suddenly become common practice? The only place a return value seems useful is when refactoring a generator and you need to mend having loss of a shared scope. What other use is there for a return value? It would seem unfortunate for it to be considered a runtime error since this would prevent sharing a generator amongst "yield from" and non-"yield from" use cases. Although, it would be trivial to do: class K: ... def _f(): yield 1 return 2 # used internally def f() # squelch the runtime error yield from self._f() As Greg has said a number of times, we allow functions to return values with them silently being ignored all the time. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tuples and underorderable types
Raymond Hettinger wrote: > Would it make sense to provide a default ordering whenever the types are > the same? > >def object.__lt__(self, other): >if type(self) == type(other): > return id(self) < id(other) >raise TypeError No. This only makes it more difficult for someone wanting to behave smartly with incomparable types. I can easily imagine someone wanting incomparable objects to be treated as equal wrt. sorting. I am thinking especially with respect to keeping the sort stable. I think many developers would be surprised to find, >>> a = >>> tasks = [(10, lambda: 0), (20, lambda: 1), (10, lambda: 2)] >>> tasks.sort() >>> assert tasks[0][1]() == 0 , is not guaranteed. Moreover, I fail to see your point in general as a bug if you accept that there is not all objects can be total ordered. We shouldn't be patching the object base class because of legacy code that relied on sorting tuples; this code should be updated to either use a key function. -Scott -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
Tarek Ziadé wrote: > On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote: >> Another general principle: even in the draft PEP, say "is", not "will >> be". > > Ok I'll fix that. That's a French stuff : in french, "will be" isn't > speculative at all. > I don't think "will be" is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, "X will be done." If "X" is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do "X". In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of "will be" in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] functools.compose to chain functions together
Greg Ewing wrote:
> Jason R. Coombs wrote:
>> I had a use case that was compelling enough that I thought there
>> should be something in functools to do what I wanted.
>
> I think this is one of those things that a small minority of
> people would use frequently, but everyone else would use
> very rarely or never. The decision on whether to include
> something in the stdlib needs to be based on the wider
> picture.
>
> In this case, it's trivial to write your own if you want
> it. As they say, "not every one-line function needs to
> be in the stdlib".
>
I have never found these arguments compelling. They are obviously not
true (e.g., itertools.compress()[1] added in 2.7/3.1), and so what I
really hear is: "I don't like it and I outrank you."
I can't help invoke part of PEP309's justification for
functools.partial()[2]:
"""
I agree that lambda is usually good enough, just not always. And I want
the possibility of useful introspection and subclassing.
"""
The same reasoning would seem to apply here. In the OP's example, the
meta-decorator becomes opaque due to the use of a lambda. If one could
introspect a compose(), then introspection tools could actually know the
set of decorators being applied. As it is, the "preferred" method of
using a lambda actually makes it quite hard to know anything.
class compose():
def __init__(self, *funcs):
if not funcs:
raise ValueError(funcs)
self.funcs = funcs
def __call__(self, *args, **kwargs):
v = self.funcs[-1](*args, **kwargs)
for func in reversed(self.funcs[:-1]):
v = func(v)
return v
meta = functools.compose(decorator_a, decorator_b)
print meta.funcs
meta = lambda f: decorator_a(decorator_b(f))
# impossible, short of disassembling the lambda
-Scott
[1] http://docs.python.org/3.1/library/itertools.html#itertools.compress
"""
def compress(data, selectors):
# compress('ABCDEF', [1,0,1,0,1,1]) --> A C E F
return (d for d, s in zip(data, selectors) if s)
"""
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Raymond Hettinger wrote: >> I was just wondering if a bytecode for a superinstruction of the common >> sequence: >> >> 6 POP_TOP >> 7 LOAD_CONST 0 (None) >> 10 RETURN_VALUE >> >> might be worth it. > > [Collin Winter] >> I doubt it. You'd save a bit of stack manipulation, but since this >> will only appear at the end of a function, I'd be skeptical that this >> would make any macrobenchmarks (statistically) significantly faster. > > I concur with Collin. And since it appears only at the end of a function, > the optimization doesn't help inner-loops in a function (where most of > the time usually spent). > I fail to understand this crude logic. How often is the inner-loop really going to solely call C code? Any call to Python in an inner-loop is going to suffer this penalty on the order of the number of loop iterations)? -Scott -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Raymond Hettinger wrote: >> I was just wondering if a bytecode for a superinstruction of the common >> sequence: >> >> 6 POP_TOP >> 7 LOAD_CONST 0 (None) >> 10 RETURN_VALUE >> >> might be worth it. > > [Collin Winter] >> I doubt it. You'd save a bit of stack manipulation, but since this >> will only appear at the end of a function, I'd be skeptical that this >> would make any macrobenchmarks (statistically) significantly faster. > > I concur with Collin. And since it appears only at the end of a function, > the optimization doesn't help inner-loops in a function (where most of > the time usually spent). > I fail to understand this crude logic. How often is the inner-loop really going to solely call C code? Any call to Python in an inner-loop is going to suffer this penalty on the order of the number of loop iterations)? -Scott -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Chris Withers wrote:
> - Apache in this instance responds with HTTP 1.1, even though the wget
> request was 1.0, is that allowed?
>
> - Apache responds with a chunked response only to httplib. Why is that?
>
I find it very confusing that you say "Apache" since your really want to
say "Coyote" which is to say "Tomcat".
http11processor.j...@1547:
if (entityBody && http11 && keepAlive) {
outputBuffer.addActiveFilter
(outputFilters[Constants.CHUNKED_FILTER]);
contentDelimitation = true;
headers.addValue(
Constants.TRANSFERENCODING).setString(Constants.CHUNKED);
} else {
outputBuffer.addActiveFilter
(outputFilters[Constants.IDENTITY_FILTER]);
}
So, as Oleg said, it's because httplib talks HTTP/1.1 and wget talks
HTTP/1.0. All HTTP/1.1 client must support chunked transfer-encoding,
and apparently Tomcat/Coyote defaults to that unless it is either an
empty message, not a HTTP/1.1 client, or the request is not to be kept
alive ("Connection: close" or no more keep-alive slots on the server).
As Simon said, changing this to do ''.join(chunks) is really the best
first step to take.
-Scott
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
R. David Murray wrote:
> On Tue, 15 Sep 2009 at 14:28, Antoine Pitrou wrote:
>> Andrew McNamara object-craft.com.au> writes:
>>>
>>>>>> ipaddr.IPv4Network('192.168.1.1/16').network
>>> IPv4Address('192.168.0.0')
>>
>> Er, does this mean that taking the `network` attribute from a network
>> object
>> actually gives an address object (not a network)?
>> It looks horribly misleading to me.
>
> That was my opinion, too. I've always called that number the 'zero'
> of the network, but other people said that it is usually called the
> 'network'.
>
> I also find it odd to see a 'network' with an IP address of 192.168.1.1.
> That looks like a host-address-plus-netmask to me, not a network
> address.
>
It just happened that I needed a tool today to calculate the gateway IP
for an interface, and I took it as an opportunity to try out this ipaddr
module. I'll attempt to relate my experience below...
I have to concur with the opinions above. I was very confused by the
following error:
>>> addr = ipaddr.IPAddress("10.1.2.3/255.255.240.0")
...
ipaddr.IPAddressIPValidationError: '98.223.189.24/255.255.240.0' is not
a valid address (hint, it's probably a network)
Because, it *is* a address of a host on a network. I gave in and said:
>>> net = ipaddr.IPNetwork("10.1.2.3/255.255.240.0")
But then, I was dumbfounded as to how I could get the gateway IP from
this IPNetwork object. It took me a while to figure out that you can
iterate over IPNetwork instances:
>>> gateway = net[1]
I was then confused, because:
>>> print(type(gateway))
Which sorta blew my mind.. I fully expected to receive an IPNetwork back
from that operation. It is unclear to me why the network information
gets chucked by that operation. I foresee having to work around that in
real applications by doing something obnoxious like:
>>> actual_gateway = ipaddr.IPNetwork("%s/%s" % (gateway, addr.netmask))
But since I only actually needed the IP address for the gateway, it
fulfilled my needs.
In the end, I found the names IPNetwork/IPAddress and their
instantiations confusing. ISTM that IPNetwork is overloaded and plays
two roles of being an IPNetwork and being an IPAddressWithNetwork. And
finally, it's unclear to me why iterating over a IPNetwork would not
produce a sequence of IPNetwork(/IPAddressWithNetwork) objects.
ISTM that if I started with an IPNetwork object, the API should always
return IPNetwork objects. If I want just an IPAddress object, then I can
always fetch the "ip" attribute to get that. Perhaps there is some
compelling conceptual argument as to why this is not correct, but it
seems like the API destroys information needlessly.
Just my opinion..
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote:
> On Tue, Sep 15, 2009 at 10:16 AM, Scott Dial
> wrote:
>> In the end, I found the names IPNetwork/IPAddress and their
>> instantiations confusing. ISTM that IPNetwork is overloaded and plays
>> two roles of being an IPNetwork and being an IPAddressWithNetwork. And
>> finally, it's unclear to me why iterating over a IPNetwork would not
>> produce a sequence of IPNetwork(/IPAddressWithNetwork) objects.
>
> What you're describing, at least for the networks, is basically what
> ipaddr was. It seemed, after much heated discussion, that the
> community felt that
>
> ipaddr.IPv4Network('1.1.1.0/24')[0]
>
> was actually an individual address. ie, each of the addresses in
> 1.1.1.0/24 are individual addresses, rather than networks with /32
> prefix lengths.
>
For clarity, I am going to call the current design "A":
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Address('1.1.1.0')
And what itt sounds like what you are describing as the old behavior is
this (design "B"):
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Network('1.1.1.0/32')
Which is different than what I was trying to describe. I expected
(design "C"):
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Network('1.1.1.0/24')
My summarization of these designs would be that "B" is wrong. And that
"A" is better than "B", certainly. At least "A" is not saying something
untrue. However, "C" would be truthful and retains a superset of
information that "A" provides (the "ip" attribute of IPNetwork).
In other words, I don't see why obtaining a host address would *not*
retain the hostmask from the network it was obtained from. I am not
disagreeing with it being an individual address. I am disagreeing that
IPNetwork itself already does represent individual addresses (hence my
aliasing it with IPAddressWithNetwork). And wrt, the logical return
would be another IPAddressWithNetwork retaining the same mask.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote:
> but that's what's being suggested here for networks.
>
>>>> ipaddr.IPv4Network('1.1.1.1/24')[0][1][2][3]...
This example here solidifies my support of RDM's suggestion of there
being 3 types:
IPv4Address
IPv4AddressWithNetwork (or as he called it: IPv4AddressWithMask)
IPv4Network
The primary difference between IPv4Network and IPv4AddressWithNetwork
would be that IPv4AddressWithNetwork would not support
iteration/indexing. The example above makes no sense logically and my
original suggestion unknowingly opened that can of worms. If indexing a
IPv4Network returned IPv4AddressWithNetwork, then that would remove that
oddity.
This would also solve the weirdness that Stephen brought up in another
branch of this discussion:
Stephen J. Turnbull wrote:
> Scott Dial writes:
> > ipaddr.IPv4Network('1.1.1.0/24')[0] ==
> > ipaddr.IPv4Network('1.1.1.0/24')
>
> So foo returns True?
>
> def foo():
> a = ipaddr.IPv4Network('1.1.1.0/24')
> return a[0] == a
>
> That seems ... weird. Maybe you committed a typo?
The root of the weirdness is that my proposition would appear to make
IPv4Network play double-duty. However, it already does! That you can
instantiate a IPv4Network object with a non-zero host is a clue, and
that it is apparently the only way to input a host with a mask is the
second clue.
If I have an IP (10.2.3.4) and I know the netmask (say, 255.255.0.0),
then how do I get the network that it is on? As it stands, the only way
I see to do this is to do:
>>> net = ipaddr.IPv4Network("10.2.3.4/255.255.0.0")
Which is strange, because I didn't input a *network*, I inputted an
*address* on a network. Moreover, it's strange because:
>>> net[0] == net.ip
False
If I needed that identity to hold true, then I have to do:
>>> net = ipaddr.IPv4Network("%s/%s" % (net[0], net.netmask)
Although it is unclear why a "network" has an "ip" attribute. As far as
I am concerned, IPv4Network objects should *only* have a net.network.
Hopefully at this point, I have made the case that IPv4Network already
is playing double-duty as a IPv4Network and IPv4AddressWithNetwork. And
that the API would be well-served by splitting that role and that it
would be considerably less confusing.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
R. David Murray wrote:
> On Tue, 15 Sep 2009 at 21:58, Antoine Pitrou wrote:
>> Le mardi 15 septembre 2009 à 15:48 -0400, R. David Murray a écrit :
>>> However, I do not think
>>> that the proposed API should accept, eg, IPv4Network('192.168.1.1/24')
>>> as valid. That's just too confusing and error prone.
>>
>> Indeed, it should throw some kind of ValueError instead.
>
> Peter, what do you think?
This change would have to be dependent on Antoine's suggestion of:
>>> addr, net = parse_address_with_mask('192.168.1.1/24')
>>> addr == IPv4Address('192.168.1.1')
True
>>> net == IPv4Network('192.168.1.0/24')
True
Otherwise, I am ok with this change too. It resloves the weird duality
of IPv4Network. And as RDM says, anyone who wants the
IPv4AddressWithNetwork functionality can just role it themselves. At the
moment of iteration, you have access to what network it is and can build
your own IPv4AddressWithNetwork objects.
net = IPv4Network('10.0.0.0/24')
netaddrs = []
for addr in net:
netaddrs.append(IPv4AddressWithNetwork(addr, net))
I guess I am ok with this. It seems sub-optimal (why not just return a
IPv4AddressWithNetwork to begin with?) but I suppose it is no less
efficient since the same objects would be constructed.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Sebastian Rittau wrote:
> On Tue, Sep 15, 2009 at 01:16:06PM -0400, Scott Dial wrote:
>>>>> net = ipaddr.IPNetwork("10.1.2.3/255.255.240.0")
>> But then, I was dumbfounded as to how I could get the gateway IP from
>> this IPNetwork object.
>
> Well, you can't. There is no way to determine a gateway, without querying
> the network topology. This is clearly outside the scope of this module.
> The only two known host addresses of a network are the network address
> (10.1.0.0 in the example you gave) and the broadcast address (10.1.15.255).
I was not writing a general utility for the world to use. On *my*
network it is always the [1]th address used for the gateway. But, I
agree in general that is not true.
>> Which sorta blew my mind.. I fully expected to receive an IPNetwork back
>> from that operation. It is unclear to me why the network information
>> gets chucked by that operation.
>
> This makes perfect sense to me. An IP network consists of a list of IP
> addresses. Returning /32 networks seems kind of pointless to me.
As I clarified later, I expected it to still be a /24 network, not a
/32. /32 makes no sense.
> But a gateway is not an IP address plus hostmask. A gateway is just a single
> IP address. What is the use of adding a hostmask to the gateway IP address
> or some other IP address inside the network?
My argument was from a destruction of information standpoint. Why is the
network chucked away? Wouldn't it be handy to be able to inspected
addr.network to see what network it was on? Perhaps because you are
building routing rules and need to actually know that. As it is, you
will just have to build your own IPAddressWithNetwork objects.
Hopefully I have covered all of you inquiries, but I feel like that a
lot of this is covered by many other replies in this thread.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote: > On Wed, Sep 16, 2009 at 8:21 PM, Andrew McNamara > wrote: >>>> I think we're in a painful middle ground now - we should either go back >>>> to the idea of a single class (per protocol), or make the distinctions >>>> clear (networks are containers and addresses are singletons). >>>> >>>> Personally, I think I would be happy with a single class (but I suspect >>>> that's just my laziness speaking). However, I think the structure and >>>> discipline of three classes (per protocol) may actually make the concepts >>>> easier to understand for non-experts. >>> I think this is where we disagree. I don't think the added complexity >>> does make it any easier to understand. >> I argue that we're not actually adding any complexity: yes, we add >> a class (per protocol), but we then merely relocate functionality to >> clarify the intended use of the classes. > > And I argue the moving this functionality to new classes (and adding > new restrictions to existing classes) doesn't buy anything in the way > of overall functionality of the module or a developer's ability to > comprehend intended uses. Speaking as the originator of this thread of discourse, the lack of a 3rd class was exactly the source of my confusion and my inability to communicate my confusion to everyone. I clearly did not understand the intended uses of the IPNetwork type because it was capable of playing two roles that are decidedly different conceptually. So, I must respectfully disagree with you. >>> You have an address + netmask. ergo, you have a Network object. >> In a common use case, however, this instance will not represent a >> network at all, but an address. It will have container-like behaviour, >> but it should not (this is a property of networks, not addresses). So >> the instance will be misnamed and have behaviours that are, at best, >> misleading. This is exactly the confusion and duality that I struggled with. >> This isn't about shortcuts, but about correctness... having the Network >> object represent a network, and having Address objects represent >> end-points, and having errors discovered as early as possible. > > Then what I don't see is the purpose of your > network-only-network-object. essentially identical functionality can > be obtained with the module as is w/o the added complexity of new > classes. The purpose is to avoid conflating IPNetwork with an IPAddress that has a mask. If the IPNetwork didn't accept a non-zero host and instead required a developer to use a helper to construct a IPNetwork with a proper address, then there would be less confusion about what exactly a IPNetwork is meant to represent. As it stands, it's purposes is muddled by accepting host addresses too. I am rather indifferent whether there needs to be a IPAddressWithMask type. If that is needed, then it is rather easy to create a type that does that. And, if it is a common pattern, then it could be added to the module later in life. -- Scott Dial [email protected] [email protected] -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Martin v. Löwis wrote:
>> Martin v. Löwis v.loewis.de> writes:
>>>> Could you explain what benefit there is for allowing the user to create
>>>> network objects that don't represent networks? Is there a use-case
>>>> where these networks-that-aren't-networks are something other than a
>>>> typo? Under what circumstances would I want to specify a network as
>>>> 192.168.1.1/24 instead of 192.168.1.0/24?
>>>>
>> [...]
>>> So Python code has to make the computation, and it seems most natural
>>> that the IP library is the piece of code that is able to compute a
>>> network out of that input.
>> The thing is, it doesn't create a network, it creates a hybrid "network plus
>> host" which retains knowledge about the original host (that is, 192.168.1.1
>> rather than simply 192.168.1.0, if you enter "192.168.1.1/24").
>>
>> That's what the OP meant with "networks-that-aren't-networks", and that's
>> what
>> people are objecting to.
>
> That's not the question that was asked, though - the question asked
> was "Under what circumstances would I want to specify...". I hope
> most people agree that it is desirable to be able to specify a network
> not just by its network address.
I can't help but feel that you and many others jumping on Antoine et al.
have skipped reading or only skimmed this topic of discussion.
Antoine Pitrou wrote:
> We just need a factory function which returns a tuple after parsing:
>
> >>> addr, net = parse_address_with_mask('192.168.1.1/24')
> >>> addr == IPv4Address('192.168.1.1')
> True
> >>> net == IPv4Network('192.168.1.0/24')
> True
I still back this suggestion. I would also support a "loose=True" flag
for the IPNetwork constructor, which would allow host bits to be non-zero.
However, I will *never* support a proposal that includes retaining the
host address. I believe I am not alone in that. However, Peter has made
it quite clear that he will *never* change that about the IPNetwork classes.
I would be quite glad to not have ipaddr included in the stdlib. As I
can only imagine code that uses it being quite confusing to maintain
(e.g., "wait is that actually a network or is that someone using
IPNetwork to be an IPAddressWithMask?").
-1.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Barry Warsaw wrote: > If not, I'll try to > spend some time over the next few days looking at outstanding bugs, and > marking release blockers, etc. I'd like to draw attention to: http://bugs.python.org/issue5949 Which is a regression of imaplib.py introduced in r57680. Ever since I switched to python 2.6 on my box, I have had issues with getmail[1] getting stuck in an infinite loop swallowing memory (note: only applies to IMAP SSL connections). While this code is present in older versions of python, it seems to have become a problem recently (2009-05-06 is the earliest report on the issue) perhaps due to a version bump of OpenSSL? I never noticed the problem in python2.5 even though the code is unchanged. Nevertheless, the code is clearly wrong and should be corrected. I would appreciate this bug being resolved before the next release as it effects me on a daily basis. I have submitted a patch, which reflects my local solution. -Scott [1] http://pyropus.ca/software/getmail/ -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Scott Dial wrote: > While this code is present in > older versions of python, it seems to have become a problem recently > (2009-05-06 is the earliest report on the issue) perhaps due to a > version bump of OpenSSL? I never noticed the problem in python2.5 even > though the code is unchanged. To answer my own question, this was introduced in r64578 in ssl.py by the addition of the suppress_ragged_eofs feature (seems to originate with issue1251). While it is a change in the behavior of the ssl module, the change falls in line with other file-like objects and their handling of EOF, so the bug still falls to imaplib. In other words, this bug appeared in 2.6 and 3.0, and is present in all subsequent versions. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Nick Coghlan wrote: > Scott Dial wrote: >> I would appreciate this bug being resolved before the next release as it >> effects me on a daily basis. I have submitted a patch, which reflects my >> local solution. > > Unfortunately, it's almost certainly too late to get this into 2.6.3. It > really needed to be brought up back when Barry called for identification > of significant 2.6 bugs and patches rather than after the RC had already > been released. > I understand. I didn't figure out the bug until after rc1 landed. It was only happening spuriously with my mail server, never when I manually tried to invoke the problem. I was forced to let my cronjob run until the kernel killed getmail for OOM, giving me the trace shown in the issue. It is impossible to break anything with this patch, as no program could proceed. The 3-line patch merely converts it back into the exception that was originally raised prior to 2.6, so it's not a new behavior in that respect either. I wouldn't have anticipated this much resistance to removing an infinite loop from the standard library. I could also for this patch from the angle that it allows a remote host the ability to execute a denial-of-service attack (even by accident!) since the infinite loop appends an empty string to a list on every loop, taking CPU time and memory with it. Allow me to be naive for a moment and say, is this not the point of rc1 but to catch bugs that should not be in the final? Of course, it's Barry's call. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] a new setuptools release?
P.J. Eby wrote:
> At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote:
>> suggest nobody hold their breath waiting for setuptools 0.7.
>
> I've never suggested or implied otherwise.
>
> But, if you like Distribute so much, why not just add it directly to the
> stdlib? ;-)
>
> There are many wins to be had from such a move, not the least of which
> is that it allows something to go into the stdlib that isn't
> (branding-wise) associated with me or setuptools, and lets it become
> owned/supported by an even wider community.
I think the biggest problem here is that the brand ("setuptools") is so
ingrained in the world that someone releasing something
setuptools-but-not-setuptools ("Distribute") is at a serious
disadvantage. Your insistence on retaining the name "setuptools" for
yourself and your vapor-ware only harms the community at-large.
Even experienced developers are unaware of Distribute's existence.. I
was entirely unaware until yourself and PJE got in a bickering exchange
on Python-Dev this past month. The extremely high visibility of your
stale package compared to their non-stale package is quite unfortunate.
You are free to say, "that is their problem," but you are not free to
say, "it is not my fault."
> AFAIK, the only reason they've had multiple releases of it is to
> address the issues with its hijacking of setuptools; in a stdlib
> version all that stuff could be dropped. Otherwise, it'd already be
> "mature".
IOW, you acknowledge that your own position and requiring them to
tolerate the parallel existence (in the world) of setuptools harms
Distribute. I fail to see how this relates to being in the stdlib. As
long as it is not called "setuptools" and packages in the world say
"import setuptools", then there are conflicts they will be forced to
managed. And moreover, as long as a stale, perhaps buggy version of
setuptools is the compatibility model they must emulate, they will have
a hard time coexisting.
> Less political bickering, and the some of the technical results I
> hoped for all along are achieved. Yay, open source.
And yet, political bickering seems to be all you are good for in this case.
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.4rc1
Barry Warsaw wrote: > 2.6.4 final is planned for 18-October. Barry, I suspect this release is primarily to quench the problems with distutils, but.. http://bugs.python.org/issue5949 doesn't seem to have been addressed by you. And this seems like it would be another unfortunate loss of an opportunity. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP about sys.implementation and implementation specific user site directory
Christian Heimes wrote: > id (required): > lower case identifier, for example "cpython", "ironpython", "jython", > "pypy" > > name (required): > mixed case name of the implementation, for example "CPython", > "IronPython", "Jython", "PyPy" Why both? Is it not true that the following is guaranteed? sys.implementation.id == sys.implementation.name.lower() Furthermore, why is a lower-case-only constant superior to a mixed-case identifier? I wonder this especially given that platform.python_implementation() already returns mixed-case constants. Why not include the version of the language? > platform (required): > platform or language of the implementation, for example "C", ".NET", > "Java" What should someone like PyPy say here? Do you have a use-case in mind? What is the relevance of this field? > runtime (required): > lower case runtime library of the implementation, for example "libc", > "msvcrt90", "java6", ".net" Same questions as above. Is the java-runtime not dependent on libc? $ ldd /opt/sun-jdk-1.6.0.15/jre/bin/java | grep libc libc.so.6 => /lib/libc.so.6 (0xb7d7) > compiler (required): > verbose name of the compiler, for example "GCC 4.3.3", "Java > HotSpot(TM) Client VM (Apple Inc.)", "Mono JIT compiler version 2.0.1" I think other commenters have already asked what exactly is considered the "compiler"? This is especially interesting if you consider PyPy, as well. What is the item being compiled that you are interested in? What is the use-case for this info? It could vary based on module: some modules could've been built with a different compiled or even compiled to bytecode by a different implementation or etc. It seems like (platform, runtime, compiler) are guesses at what *might* be useful. But, in the absence of use-cases they are not useful, perhaps misguided, and create a maintenance burden for the future. A lot of this info is available via the platform module. And the only reason given to not use the platform module was to avoid adding dependencies for site.py. I don't see that as an open invitation to adding other fields that are already available via the platform module. Until they are critical to run-time performance, why not wait to add these extra things? The only thing that has been indicated as needed is the identifier for the python implementation. sys.vm or sys.implementation may very well fully support the use cases given merely by being a string. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL behaviour under Windows
Antoine Pitrou wrote: > Could you try ccbench (*) under Windows? The only Windows system I have here > is > a qemu virtual machine and it wouldn't be very realistic to do concurrency > measurements on it. > > (*) http://svn.python.org/view/sandbox/trunk/ccbench/ > I don't really know how this test works, so I won't claim to understand the results either. However, here you go: C:\>systeminfo OS Name: Microsoft Windows XP Professional OS Version:5.1.2600 Service Pack 3 Build 2600 C:\>c:\Python26\python.exe Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32 C:\>start /B /HIGH c:\Python26\python.exe c:\ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 377 iterations/s. threads=2: 376 ( 99 %) threads=3: 380 ( 100 %) threads=4: 376 ( 99 %) regular expression (C) threads=1: 222 iterations/s. threads=2: 213 ( 95 %) threads=3: 223 ( 100 %) threads=4: 218 ( 97 %) bz2 compression (C) threads=1: 324 iterations/s. threads=2: 324 ( 99 %) threads=3: 327 ( 100 %) threads=4: 324 ( 100 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL behaviour under Windows
Antoine Pitrou wrote: > Interesting results. I wonder what they would be like on a multi-core > machine. The GIL seems to behave perfectly on your setup (no performance > degradation due to concurrency, and zero latencies). You are correct, my machine is a single-core system. I don't have any multi-core systems around to test it on, I'm still in the stone age. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
Curt Hagenlocher wrote: > But it makes more sense to > understand why someone chose to implement time.clock() on Windows the > way they did -- this seems very broken to me, and I think it should be > changed. Some SVN detective work takes this to all the way back to r7713 (1997-04-02). The original implementation checked by Guido and attributed to Mark Hammond. So, we should ask Mark why he did that. Can anyone honestly use it, as it is, without already having normalize it across platforms themselves? I don't know how much of an impact it is, but the current implementation of clock() does not require argument parsing, so the proposal to add a "absolute" boolean-flag argument is perhaps bad. This is generally a function used for performance timing and that proposal adds some amount of latency to the query. The proposal to add a clockbase() function is perhaps better because of this, you need only call it once, and you can cache the result for the life of your process. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3003 - Python Language Moratorium
Guido van Rossum wrote: > On Sun, Nov 8, 2009 at 3:06 PM, Steven D'Aprano wrote: >> No new language features in odd-numbered point releases (3.3, 3.5, ...). >> Even-numbered point releases (3.4, 3.6, ...) may include new language >> features provided they meet the usual standards for new features. > > Oh no, not the eve/odd numbering scheme. Nobody will be able to > remember which is which. > In this case, does it really matter? Usually the confusion is with stable/unstable being even/odd or odd/even, but in this case the options are stable/stable (it just happens that certain pairwise versions are related). -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7 alpha 1
Benjamin Peterson wrote: > On behalf of the Python development team, I'm pleased to announce the first > alpha release of Python 2.7. Was there a call for bugs that should be reviewed for this release? I once again will prod for attention to this show-stopping bug for using IMAP-SSL: http://bugs.python.org/issue5949 Title: IMAP4_SSL spin because of SSLSocket.suppress_ragged_eofs -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2
On 12/28/2009 5:42 AM, Martin v. Löwis wrote: >>> So specifying 2.5 would be a short-hand for *what*? >> >> 2.5 would be a shorthand for 2.5.x. So, equivalent to : >=2.5.0, < 2.6.0 > > Ok, so it's not a shorthand for a single operator anymore, but for a > more complex term. Fine with me. > >> 2.5.0 would be the notation required to describe this specific micro version. > > I think it would be a shorthand for >=2.5.0, <2.5.1, right? Actually, the logical extension is that 2.5.0 is shorthand for >=2.5.0.0, <2.5.1.0, I believe. PEP 386 versions can have an indefinite number of extradecimal versions. Pedantically, -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Download Page - AMD64
On 1/13/2010 9:04 AM, Nick Coghlan wrote: > As Windows doesn't run on non-x86 architectures, their naming is > generally just Windows (32 bit) and Windows (64 bit). That is not correct. There are IA-64 versions of Window Server. >From [1]: "Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures." So to echo what Michael said, the Microsoft nomenclature is "x64" regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by "x64" since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see "IA64" and not "x64", and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between "IA64" and "x64". I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. > Linux, on the other hand, can run on multiple 64 bit architectures, so > being more specific and using the official AMD64 nomenclature makes > sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/21/2010 1:09 PM, Hanno Schlichting wrote: > - Who holds the copyright, is there a foundation or is there a risk of > the project getting into trouble because of copyright issues? > - What licence is the code and the tools under and what affect does > that have on the code generated by the JIT compiler? http://llvm.org/docs/DeveloperPolicy.html#clp """ We intend to keep LLVM perpetually open source and to use a liberal open source license. The current license is the University of llinois/NCSA Open Source License[1], which boils down to this: * You can freely distribute LLVM. * You must retain the copyright notice if you redistribute LLVM. * Binaries derived from LLVM must reproduce the copyright notice (e.g. in an included readme file). * You can't use our names to promote your LLVM derived products. * There's no warranty on LLVM at all. We believe this fosters the widest adoption of LLVM because it allows commercial products to be derived from LLVM with few restrictions and without a requirement for making any derived works also open source (i.e. LLVM's license is not a "copyleft" license like the GPL). We suggest that you read the License[1] if further clarification is needed. """ """ We have no plans to change the license of LLVM. If you have questions or comments about the license, please contact the LLVM Oversight Group[2]. """ [1] http://www.opensource.org/licenses/UoI-NCSA.php [2] [email protected] -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)
On 1/22/2010 2:04 PM, M.-A. Lemburg wrote: > Karen Tracey wrote: >> On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord >> wrote: >>> The encoding I'm talking about is the >>> encoding that Python uses to decode a file (or encode a string) when you do >>> the following in Python 3: >>> >>>text = open(filename).read() >>>open(filename, 'w').write(some_string) >>> >>> It isn't the default encoding (always utf-8 by default in Python 3 >>> apparently), it isn't the file system encoding which is the system encoding >>> used for file names. What is the correct terminology for this platform >>> dependent encoding that Python uses here? >>> >>> >> The doc here: >> http://docs.python.org/3.1/library/functions.html?highlight=open#open just >> calls it default encoding and clarifies that is "whatever >> locale.getpreferredencoding() returns". > > ... which is a pretty poor guess, since the locale setting > is usually set to what the user wants to see in user interfaces, > not what the application want to use as file content. [...] > Applications should always provide a well-defined encoding > for use with files - they know best what encoding to use and > they also know best when to apply guessing and which set > of encodings to use as basis for such guessing. This all begs the question: why is there a default? and why is the default a guess? I have to admit that I was completely oblivious to this potential pitfall, and mostly that's because in the most common case, I am working with ASCII files. It's just serendipity that most systems specify (if not require) the locale encoding be an ASCII superset. """ locale.getpreferredencoding([do_setlocale]) Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess. """ I already know that this suggestion will not get any following because, for most people, it just works. However: "In the face of ambiguity, refuse the temptation to guess." Would it really be that unfortunate to force everyone to reconsider what they are doing when they open() files? -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
On 1/28/2010 11:57 PM, Steve Howell wrote: > --- On Thu, 1/28/10, Josiah Carlson wrote: >> [...] in the decade+ that I've been using >> Python and >> needed an ordered sequence; lists were the right solution >> 99% of the >> time [...] > > What do you think of LISP, and "car" in particular (apart from the stupidly > cryptic name)? A LISP list/pair has nothing to do with a Python list. The list of LISP is a *singly-linked* list. You cannot O(1) index a LISP list. A Python list is more equivalent to a LISP vector, which "car" does not work with; in fact, there is not even a "pop" operator -- all size changes of a vector O(n) unless the implementation is playing games (like the one you are proposing for the start and the one Python already uses for the end of a list). (And with this, clearly uninformed reply by you, I am now ignoring your trolling.) -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 8:43 AM, Cesare Di Mauro wrote: > If you use Mercurial, you can grab a local copy this way: > > hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10 > > Wpython is intended to run on any platform where CPython 2.6.4 runs. > Cesare, just FYI, your Hg repository has lost the execute bits on some files (namely "./configure" and "./Parser/asdl_c.py"), so it does not quite build out-of-the-box. I took the liberty of cloning your repo into my laptop's VirtualBox instance of Ubuntu. I ran the default performance tests from the U-S repo, with VirtualBox at highest priority. As a sanity check, I ran it against the U-S trunk. I think the numbers speak for themselves. $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../wpython2-wpython10/python Report on Linux narf-ubuntu 2.6.31-16-generic #53-Ubuntu SMP Tue Dec 8 04:01:29 UTC 2009 i686 Total CPU cores: 1 ### 2to3 ### Min: 21.629352 -> 20.893306: 1.0352x faster Avg: 22.245390 -> 21.061316: 1.0562x faster Significant (t=4.416683) Stddev: 0.58810 -> 0.11618: 5.0620x smaller Timeline: http://tinyurl.com/yawzt5z ### django ### Min: 1.105662 -> 1.115090: 1.0085x slower Avg: 1.117930 -> 1.131781: 1.0124x slower Significant (t=-11.024923) Stddev: 0.00729 -> 0.01023: 1.4027x larger Timeline: http://tinyurl.com/ydzn6e6 ### nbody ### Min: 0.535204 -> 0.559320: 1.0451x slower Avg: 0.558861 -> 0.572902: 1.0251x slower Significant (t=-7.484374) Stddev: 0.01309 -> 0.01344: 1.0272x larger Timeline: http://tinyurl.com/ygjnh5x ### slowpickle ### Min: 0.788558 -> 0.757067: 1.0416x faster Avg: 0.799407 -> 0.774368: 1.0323x faster Significant (t=12.772246) Stddev: 0.00686 -> 0.01836: 2.6759x larger Timeline: http://tinyurl.com/y8g3zjg ### slowspitfire ### Min: 1.200616 -> 1.218915: 1.0152x slower Avg: 1.229028 -> 1.255978: 1.0219x slower Significant (t=-8.165772) Stddev: 0.02133 -> 0.02519: 1.1808x larger Timeline: http://tinyurl.com/y9gg2x5 ### slowunpickle ### Min: 0.355483 -> 0.347013: 1.0244x faster Avg: 0.369828 -> 0.359714: 1.0281x faster Significant (t=6.817449) Stddev: 0.01008 -> 0.01089: 1.0804x larger Timeline: http://tinyurl.com/ybf3qg9 ### spambayes ### Min: 0.316724 -> 0.314673: 1.0065x faster Avg: 0.327262 -> 0.332370: 1.0156x slower Significant (t=-3.690136) Stddev: 0.00598 -> 0.01248: 2.0876x larger Timeline: http://tinyurl.com/ydck59l $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../unladen-swallow/python ### 2to3 ### Min: 24.833552 -> 24.433527: 1.0164x faster Avg: 25.241577 -> 24.878355: 1.0146x faster Not significant Stddev: 0.39099 -> 0.28158: 1.3886x smaller Timeline: http://tinyurl.com/yc7nm79 ### django ### Min: 1.153900 -> 0.892072: 1.2935x faster Avg: 1.198777 -> 0.926776: 1.2935x faster Significant (t=61.465586) Stddev: 0.03914 -> 0.02065: 1.8949x smaller Timeline: http://tinyurl.com/ykm6lnk ### nbody ### Min: 0.541474 -> 0.307949: 1.7583x faster Avg: 0.564526 -> 0.327311: 1.7247x faster Significant (t=57.615664) Stddev: 0.01784 -> 0.03711: 2.0798x larger Timeline: http://tinyurl.com/ylmezw5 ### slowpickle ### Min: 0.832452 -> 0.607266: 1.3708x faster Avg: 0.860438 -> 0.651645: 1.3204x faster Significant (t=20.779110) Stddev: 0.01559 -> 0.09926: 6.3665x larger Timeline: http://tinyurl.com/yaktykw ### slowspitfire ### Min: 1.204681 -> 1.038169: 1.1604x faster Avg: 1.236843 -> 1.085254: 1.1397x faster Significant (t=20.203736) Stddev: 0.02417 -> 0.07103: 2.9391x larger Timeline: http://tinyurl.com/ykgmop5 ### slowunpickle ### Min: 0.374148 -> 0.279743: 1.3375x faster Avg: 0.398137 -> 0.315630: 1.2614x faster Significant (t=16.069155) Stddev: 0.01333 -> 0.04958: 3.7203x larger Timeline: http://tinyurl.com/y9b5rza ### spambayes ### Min: 0.330391 -> 0.302988: 1.0904x faster Avg: 0.349153 -> 0.394819: 1.1308x slower Not significant Stddev: 0.01158 -> 0.35049: 30.2739x larger Timeline: http://tinyurl.com/ylq8sef -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On 1/31/2010 2:04 PM, Raymond Hettinger wrote: > On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > > It would be nice if all the compilation files could be tucked > into one single zipfile per directory to reduce directory clutter. > > It has several benefits besides tidiness. It hides the implementation > details of when magic numbers get shifted. And it may allow faster > start-up times when the zipfile is in the disk cache. > On a whim, I implemented a PEP302 loader that cached any important that it could find in sys.path into a zip file. I used running bzr as a startup benchmark, and I did my best to ensure an empty cache by running "sync; echo 3 > /proc/sys/vm/drop_caches; time bzr". On my particular machine, the "real" time was at minimum 3.5 seconds without using my ZipFileCacheLoader. With the loader, I found the same was true. The average performance was all over the place (due everything else in the operating system trying to fetch from the disk), and I lack enough data points to reach statistical significance. However, if the ".pyr" zip file is going to contain many versions of the same module, then the performance impact could be more real, since you would be forced to pull from disk *all* of the versions of a given module. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7b1 and argparse's version action
On 4/18/2010 9:44 PM, Steve Holden wrote: > Tobias Herp wrote: >> Steven Bethard schrieb: >>> On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp wrote: >>>> Steven Bethard schrieb: >>>>> But I'd really like a consensus about the correct behavior, and so far >>>>> I have not seen that. >> >> Do you take your own poll seriously? >> > When was this ever a democracy? Is consensus superficial? -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage.
On 4/26/2010 7:24 AM, Antoine Pitrou wrote: >> I think it is very much in the interest of Python to evolve our >> processes in order to encourage more core-developers. Evolving means >> experimenting *and* being willing to change. It is certainly less >> *effort* to accept the status quo, but with several more committed and >> enthusiastic (and good) core developers there is an awful lot (more) we >> could achieve. > > I certainly agree we should try to attract more good-willed and > competent contributors. > > I also agree with Stephen that, in a project with a non-trivial amount > of complexity such as CPython, not having (tracker or commit) privileges > is not the real barrier to entry. You have to learn how the software > works, how its development works, who are the people working on it, etc. > I'd like to respond to Michael's comment about the "possibly hundreds of modules in the standard library without a maintainer." My own experience (issue5949) has been positive despite the lack of a dedicated maintainer. When I had my own itch to scratch, nobody stopped me from scratching it.. some people told me when I could scratch it and how they'd like it scratched.. but I wasn't ignored or rejected despite the lack of a maintainer. Thanks to RDM for giving my issue attention. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/26/2010 8:03 PM, Nick Coghlan wrote: > On 27/05/10 02:27, Terry Reedy wrote: >> I am suggesting that if we add a package, we do it right, from the >> beginning. > > This is a reasonable point of view, but I wouldn't want to hold up PEP > 3148 over it (call it a +0 for the idea in general, but a -1 for linking > it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a "concurrent" namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it "futures", without putting it in a "concurrent" namespace. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 5/27/2010 7:14 AM, Colin H wrote:
> def define_stuff(user_code):
> context = {...}
> stuff = {}
> stuff.update(context)
>
> exec(user_code, stuff)
>
> return_stuff = {}
> return_stuff.update(stuff)
>
> del return_stuff['__builtins__']
> for key in context:
> if key in return_stuff and return_stuff[key] == context[key]:
> del return_stuff[key]
>
> return return_stuff
I'm not sure your application, but I suspect you would benefit from
using an identity check instead of an __eq__ check. The equality check
may be expensive (e.g., a large dictionary), and I don't think it
actually is checking what you want -- if the user_code generates an
__eq__-similar dictionary, wouldn't you still want that? The only reason
I can see to use __eq__ is if you are trying to detect user_code
modifying an object passed in, which is something that wouldn't be
addressed by your original complaint about exec (as in, modifying a
global data structure).
Instead of:
> if key in return_stuff and return_stuff[key] == context[key]:
Use:
> if key in return_stuff and return_stuff[key] is context[key]:
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/27/2010 4:13 AM, Brian Quinlan wrote: > On 27 May 2010, at 17:53, Floris Bruynooghe wrote: >> On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: >>> On 27/05/10 00:31, Brian Quinlan wrote: >>>> You have two semantic choices here: >>>> 1. let the interpreter exit with the future still running >>>> 2. wait until the future finishes and then exit >>> >>> I'd go for (1). I don't think it's unreasonable to >>> expect a program that wants all its tasks to finish >>> to explicitly wait for that to happen. >> >> I'd got for (1) as well, it's no more then reasonable that if you want >> a result you wait for it. And I dislike libraries doing magic you >> can't see, I'd prefer if I explicitly had to shut a pool down. >> >> I'm glad I'm not alone in preferring (1) tough. > > Keep in mind that this library magic is consistent with the library > magic that the threading module does - unless the user sets > Thread.daemon to True, the interpreter does *not* exit until the thread > does. Given your rationale, I don't understand from the PEP: > shutdown(wait=True) > > Signal the executor that it should free any resources that it is > using when the currently pending futures are done executing. Calls to > Executor.submit and Executor.map and made after shutdown will raise > RuntimeError. > > If wait is True then the executor will not return until all the > pending futures are done executing and the resources associated with > the executor have been freed. Can you tell me what is the expected execution time of the following: >>> executor = ThreadPoolExecutor(max_workers=1) >>> executor.submit(lambda: time.sleep(1000)) >>> executor.shutdown(wait=False) >>> sys.exit(0) I believe it's 1000 seconds, which seems to defy my request of shutdown(wait=False) because "secretly" the Python exit is going to wait anyways. ISTM, it is much easier to get behavior #2 if you have behavior #1, and it would also seem rather trivial to make ThreadPoolExecutor take an optional argument specifying which behavior you want. Your reference implementation does not actually implement the specification given in the PEP, so it's quite impossible to check this myself. There is no wait=True option for shutdown() in the reference implementation, so I can only guess what that implementation might look like. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 5:09 PM, Barry Warsaw wrote: >> What use case does this address? > > Specifically, it's the use case where we (Debian/Ubuntu) plan on installing > all Python 3.x packages into /usr/lib/python3/dist-packages. As of PEP 3147, > we can do that without collisions on the pyc files, but would still have to > symlink for extension module .so files, because they are always named foo.so > and Python 3.2's foo.so won't (modulo PEP 384) be compatible with Python 3.3's > foo.so. If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared)? > So using the same trick as in PEP 3147, if we can name Python 3.2's foo > extension differently than the incompatible Python 3.3's foo extension, we can > have them live in the same directory without symlink tricks. Why would a symlink trick even be necessary if there is a version-unspecific directory and a version-specific directory on the search path? >> PEP 3147 addresses the fact that the user may have different versions of >> Python installed and each wants to write a .pyc file when loading a module. >> .so files are not generated simply by running the Python interpreter, ergo >> .so files are not an issue for that use case. > > See above. It doesn't matter whether the pyc or so is created at run time by > the user or by the distro build system. If the files for different Python > versions end up in the same directory, they must be named differently too. But the only motivation for doing this with .pyc files is that the .py files are able to be shared, since the .pyc is an on-demand-generated, version-specific artifact (and not the source). The .so file is created offline by another toolchain, is version-specific, and presumably you are not suggesting that Python generate it on-demand. > >> If you want to make it so a system can install a package in just one >> location to be used by multiple Python installations, then the version >> number isn't enough. You also need to distinguish debug builds, profiling >> builds, Unicode width (see issue8654), and probably several other >> ./configure options. > > This is a good point, but more easily addressed. Let's say a distro makes > three Python 3.2 variants available, one "normal" build, a debug build, and > UCS2 and USC4 versions of the above. All we need to do is choose a different > .so ABI tag (see previous follow) for each of those builds. My updated patch > (coming soon) allows you to define that tag to configure. So e.g. Why is this use case not already addressed by having independent directories? And why is there an incentive to co-mingle these version-punned files with version-agnostic ones? > Mix and match for any other build options you care about. Because the distro > controls how Python is configured, this should be fairly easy to achieve. For packages that have .so files, won't the distro already have to build multiple copies of that package for all version of Python? So, why can't it place them in separate directories that are version-specific at that time? This is not the same as placing .py files that are version-agnostic into a version-agnostic location. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 8:23 PM, James Y Knight wrote: > On Jun 24, 2010, at 5:53 PM, Scott Dial wrote: >> If the package has .so files that aren't compatible with other version >> of python, then what is the motivation for placing that in a shared >> location (since it can't actually be shared) > > Because python looks for .so files in the same place it looks for the > .py files of the same package. My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk. Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme). Either the motivation for this PEP is inaccurate or I am failing to understand how this is *simpler*. In the case of pure-python, this PEP is clearly a win, but I have not seen an argument that it is a win for .so files. Moreover, the PEP itself is titled "PYC Repository Directories" (not "shared site-packages") and makes no mention of .so files at all. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 9:18 PM, Greg Ewing wrote: > Scott Dial wrote: > >> But the only motivation for doing this with .pyc files is that the .py >> files are able to be shared, > > In an application made up of a mixture of pure Python and > extension modules, the .py files are able to be shared too. > Seems to me that a similar motivation exists here as well. > Not exactly the same, but closely related. > If I recall Barry's motivation correctly, the PEP was intended to simplify the installation of packages for multiple versions of Python, although the PEP states that in a less direct way. In the case of pure-python packages, this is merely about avoiding .pyc collisions. But, in the case of packages with .so files, I fail to see how this is simpler (in face, I believe it to be more complicated). So, I am not sure the PEP supports this feature being proposed (since it makes no mention of .so files), and more importantly, I am not sure it actually makes anything better for anyone (still requires multiple compilations and un/install gymnastics). -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/25/2010 2:58 PM, Brett Cannon wrote: > I assume you are talking about PEP 3147. You're right that the PEP was > for pyc files and that's it. No one is talking about rewriting the > PEP. Yes, I am making reference to PEP 3147. I make reference to that PEP because this change is of the same order of magnitude as the .pyc change, and we asked for a PEP for that, and if this .so stuff is an extension of that thought process, then it should either be reflected by that PEP or a new PEP. > The motivation Barry is using is an overarching one of distros > wanting to use a single directory install location for all installed > Python versions. That led to PEP 3147 and now this work. It's unclear to me that that is the correct motivation, which you are divining. As I understand it, the motivation to be to *simplify installation* for distros, which may or may not be achieved by using a single directory. In the case of pure-python packages, a single directory is an obvious win. In the case of mixed-python packages, I remain to be persuaded there is any improvement achieved. > This is meant to be used by distros in a programmatic fashion, so my > response is "so what?" Their package management system is going to > maintain the directory, not a person. Then why is the status quo unacceptable? I have already explained how this will still require programmatic steps of at least the same difficulty as the status quo requires, so why should we change anything? I am skeptical that this is a simple programmatic problem either: take any random package on PyPI and tell me whether or not it has a .so file that must be compiled. If such a .so file exists, then this package must be special-cased and compiled for each version of Python on the system (or will ever be on the system?). Such a package yields an arbitrary number of .so files due to the number of version of Python on the machine, and I can't imagine how it is simpler to manage all of those files than it is to manage multiple site-packages. > You're conflating what is being discussed with PEP 3147. That PEP is > independent of this. PEP 3147 just empowered this work to be relevant. Without a PEP (be it PEP 3147 or some other), what is the justification for doing this? The burden should be on "you" to explain why this is a good idea and not just a clever idea. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/26/2010 4:06 PM, Matthias Klose wrote: > On 25.06.2010 22:12, James Y Knight wrote: >> On Jun 25, 2010, at 4:53 AM, Scott Dial wrote: >>> Placing .so files together does not simplify that install process in any >>> way. You will still have to handle such packages in a special way. >> >> This is a good point, but I think still falls short of a solution. For a >> package like lxml, indeed you are correct. Since debian needs to build >> it once per version, it could just put the entire package (.py files and >> .so files) into a different per-python-version directory. > > This is what is currently done. This will increase the size of packages > by duplicating the .py files, or you have to install the .py in a common > location (irrelevant to sys.path), and provide (sym)links to the > expected location. "This is what is currently done" and "provide (sym)links to the expected location" are conflicting statements. If you are symlinking .py files from a shared location, then that is not the same as "just install the package into a version-specific location". What motivation is there for preferring symlinks? Who cares if a ditro package install yields duplicate .py files? Nor am I motivated by having to carry duplicate .py files in a distribution package (I imagine the compression of duplicate .py files is amazing). > A "different per-python-version directory" also has the disadvantage > that file conflicts between (distribution) packages cannot be detected. Why? That sounds like a broken tool, maybe I am naive, please explain. If two packages install /usr/lib/python2.6/foo.so that should be just as detectable two installing /usr/lib/python-shared/foo.cpython-26.so If you *must* compile .so files for every supported version of python at packaging time, then you are already saying the set of python versions is known. I fail to see the difference between a package that installs .py and .so files into many directories than having many .so files in a single directory; except that many directories *already* works. The only gain I can see is that you save duplicate .py files in the package and on the filesystem, and I don't feel that gain alone warrants this fundamental change. I would appreciate a proper explanation of why/how a single directory is better for your distribution. Also, I haven't heard anyone that wasn't using debian tools chime in with support for any of this, so I would like to know how this can help RPMs and ebuilds and the like. > I don't think that installation into different locations based on the > presence of extension will work. Should a location really change if an > extension is added as an optimization? Splitting a (python) package > into different installation locations should be avoided. I'm not sure why changing paths would matter; any package that writes data in its install location would be considered broken by your distro already, so what harm is there in having the packaging tool move it later? Your tool will remove the old path and place it in a new path. All of these shenanigans seem to manifest from your distro's python-support/-central design, which seems to be entirely motivated by reducing duplicate files and *not* simplifying the packaging. While this plan works rather well with .py files, the devil is in the details. I don't think Python should be getting involved in what I believe is a flawed design. What happens to the distro packaging if a python package splits the codebase between 2.x and 3.x (meaning they have distinct .py files)? As someone else mentioned, how is virtualenv going to interact with packages that install like this? -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/30/2010 2:53 PM, Barry Warsaw wrote: > It might be amazing, but it's still a significant overhead. As I've > described, multiply that by all the py files in all the distro packages > containing Python source code, and then still try to fit it on a CDROM. I decided to prove to myself that it was not a significant issue to have parallel directory structures in a .tar.bz2, and I was surprised to find it much worse at that then I had imagined. For example, # cd /usr/lib/python2.6/site-packages # tar --exclude="*.pyc" --exclude="*.pyo" \ -cjf mercurial.tar.bz2 mercurial # du -h mercurial.tar.bz2 640Kmercurial.tar.bz2 # cp -a mercurial mercurial2 # tar --exclude="*.pyc" --exclude="*.pyo" \ -cjf mercurial2.tar.bz2 mercurial mercurial2 # du -h mercurial.tar.bz2 1.3Mmercurial2.tar.bz2 So, I was definitely wrong in saying that you do better than doubling. >> What happens to the distro packaging if a python package splits the >> codebase between 2.x and 3.x (meaning they have distinct .py files)? > > The Debian/Ubuntu approach to Python 2/3 support is to provide them in > separate distro packages. E.g. for Python package foo, you would have Debuntu > package python-foo (for the Python 2.x version) and python3-foo. We do not > share source between Python 2 and 3 versions, at least not yet . I had asked this question to point out that you will still need to accommodate some form of version-specific packages (I am not a debuntu expert by any means). And, I think your response is an acknowledgment of that fact, however it's certainly true that there are few examples, as you said. I appreciate all your replies. I am not sure a PEP is really needed here, but to having had all of this discussed and explained on the mailing list is certainly useful. I trust that yourself and the debuntu python group will end up chasing down and taking care of any quirks that this change might cause, so I am not worried about it. :D -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Return from generators in Python 3.2
On 8/26/2010 11:00 AM, Yury Selivanov wrote: > If we decide to postpone this feature till Python 3.3, than we'll push it all > back > The change is tiny, but it means really a lot. AFAICT, this change was the most controversial part of PEP 380. > PS I'm attaching a patch to the letter; it's far from ideal state, but > contains the > GeneratorReturn exception, code to raise it and the corresponding unittests. I believe overloading StopIteration for this purpose was considered more acceptable than creating a new exception. BTW, attaching patches to emails on this list is generally the best way to have few look at your patch. :-p Also, this seems more appropriate for python-ideas. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/25/2010 5:37 PM, Martin v. Löwis wrote: >> Unfortunately, most sites using OpenID seem have an awkward login >> process. Maybe it's just me (I don't use OpenID much) but I expect >> that without a lot more handholding of new users, OpenID actually >> turns more people away than any other registration/login process. > > So how do you like the OpenID login of PyPI, which has a Google, > MyOpenID and Launchpad icon, which users need to click on to create > in account or login? > > The ultra geeks demanded and got a separate page where they > can enter long URLs. Having just tried this out. A few comments: 1) Registering via OpenID is a bit clumsy since there is a "Register" link that does not mention OpenID. 2) The URL registered with the OpenID provider is a bit of a wart: "http://pypi.python.org/pypi?:action=openid_return"; vs. "http://bitbucket.org/"; 3) The email I received asked me to "Complete your Cheese Shop registration" which I think is just an oversight since the relabeling to pypi. 4) It's a bit clumsy that "Login" pops up an HTTP Authentication prompt, which is useless to someone who only has never set a password and relies only on an OpenID credential. Furthermore, the 401 page does not provide a quick way to get to use OpenID. In general, I am pretty happy with pypi's support of OpenID considering it allowed me to use my own provider, which often has not been the case with other sites. My experience with trying to adopt OpenID as a way of life has been poor mostly because many sites fail to support anything but a few major OpenID providers (e.g., Google). I appreciate has a fast-path for those providers and yet let's me still use my own. Although, I think it would be nice if I didn't have to go to another page to do that, but I may be biased by having such a short OpenID URI. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/26/2010 3:12 AM, Martin v. Löwis wrote: >> 2) The URL registered with the OpenID provider is a bit of a wart: >> "http://pypi.python.org/pypi?:action=openid_return"; vs. >> "http://bitbucket.org/"; > > You mean, as this is what the provider then shows you for confirmation? The provider also lists the trusted sites by these return URLs, and that is where I saw it as being a bit of a wart. I use the OpenID plugin for WordPress as my provider, so it may be that it doesn't do this correctly. I noticed that Google shows just "pypi.python.org", but the WordPress plugin shows that return URL instead. Nevertheless, I agree that it's too late/not worth it to change that now. > I think there is no way out wrt. to the basic auth prompt. I could > label the "Login" link "Password login" if you think this would help. The basic auth prompt doesn't bother me so much as the fact that the 401 doesn't have a "Use OpenID [Google] [myOpenID] [Launchpad]" set of links; you have to use the brower's back button because the only links offered are to register or reset your password. > Preventing the browser from prompting the user on the chance they > might want to enter an OpenID is not possible, and stopping to use > basic authentication is not feasible. In theory, you could catch usernames that started with "http://";, but I imagine that only "ultra geeks" know their URIs (I have no idea what the URI for a Google account is). But, I don't see this as being worthwhile either; I just think it would be nice if the 401 page gave a quick way to correct one's mistake that didn't involve the back button. > And again, enjoying a short OpenID URI > probably does put you in the "ultra geek" category (which I > seriously don't mean as an offense). No offense taken. :) -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/26/2010 11:45 PM, R. David Murray wrote: > On Sun, 26 Sep 2010 21:56:20 -0400, Scott Dial > wrote: >> On 9/26/2010 3:12 AM, Martin v. Loewis wrote: >>> Preventing the browser from prompting the user on the chance they >>> might want to enter an OpenID is not possible, and stopping to use >>> basic authentication is not feasible. >> >> In theory, you could catch usernames that started with "http://";, but I > > No, Martin really meant "not possible": once basic auth is started, > the browser prompts for username and password and you are in basic-auth > land thereafter; the web server has *no way* to tell the browser to > *stop* using basic auth. I agree that once you reply with a 401 that you will prompt the user, but my point was what "username" means in the Authorization header is open to interpretation by the HTTP server and/or script handling the GET request. >> imagine that only "ultra geeks" know their URIs (I have no idea what the >> URI for a Google account is). But, I don't see this as being worthwhile > > Well, my OpenId is 'david.bitdance.com', so even if you could get around > the basic auth problem, looking for "http://"; wouldn't work. That's actually not a valid OpenID[1], but the OpenID specification says a relaying party "MUST" normalize identifiers[2] (in this case, prepending the "http://";). I believe bugs.python.org does this by checking for a username first(?), and failing any matches, it normalizes it for OpenID discovery. Otherwise, I can always use the canonical form of my ID "http://scottdial.com"; to login (assuming ':' and '/' are illegal characters for usernames). I say all this not with the intent of saying pypi *needs* this, but to refute the notion that OpenID must be clumsy to use. [1] http://openid.net/specs/openid-authentication-2_0.html """ Identifier: An Identifier is either a "http" or "https" URI, (commonly referred to as a "URL" within this document), or an XRI (Reed, D. and D. McAlpin, “Extensible Resource Identifier (XRI) Syntax V2.0,” .) [XRI_Syntax_2.0]. """ [2] http://openid.net/specs/openid-authentication-2_0.html#normalization """ 3. Otherwise, the input SHOULD be treated as an http URL; if it does not include a "http" or "https" scheme, the Identifier MUST be prefixed with the string "http://";. If the URL contains a fragment part, it MUST be stripped off together with the fragment delimiter character "#". See Section 11.5.2 (HTTP and HTTPS URL Identifiers) for more information. """ -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch making the current email package (mostly) support bytes
On 10/2/2010 7:00 PM, R. David Murray wrote:
> The clever hack (thanks ultimately to Martin) is to accept 8bit data
> by encoding it using the ASCII codec and the surrogateescape error
> handler.
I've seen this idea pop up in a number of threads. I worry that you are
all inventing a new kind of dual that is a direct parallel to Python 2.x
strings. That is to say,
3.x>>> b = b'\xc2\xa1'
3.x>>> s = b.decode('utf8')
3.x>>> v = b.decode('ascii', 'surrogateescape')
, where s and v should be the same "thing" in 3.x but they are not due
to an encoding trick. I believe this trick generates more-or-less the
same issues as strings did in 2.x:
2.x>>> b = '\xc2\xa1'
2.x>>> s = b.decode('utf8')
2.x>>> v = b
Any reasonable 2.x code has to guard on str/unicode and it would seem in
3.x, if this idiom spreads, reasonable code will have to guard on
surrogate escapes (which actually seems like a more expensive test). As in,
3.x>>> print(v)
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc2' in
position 0: surrogates not allowed
It seems like this hack is about making the 3.x unicode type more like
the 2.x string type, and I thought we decided that was a bad idea. How
will developers not have to ask themselves whether a given string is a
"real" string or a byte sequence masquerading as a string? Am I missing
something here?
--
Scott Dial
[email protected]
[email protected]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r85838 - python/branches/py3k/.gitignore
On 10/27/2010 3:25 AM, anatoly techtonik wrote: > On Tue, Oct 26, 2010 at 3:51 PM, Barry Warsaw wrote: >> On Oct 26, 2010, at 09:19 PM, Nick Coghlan wrote: >> >>> On Tue, Oct 26, 2010 at 5:31 PM, Georg Brandl wrote: >>>> This looks more like "Add gitignore". Do we really want to check in >>>> ignore files for every possible DVCS? >>> >>> No, but supporting the current big four open source ones (svn, hg, >>> bzr, git) seems reasonable enough. >> >> +1. A couple of extra dot files never hurt anyone. :) > > Why hg and bzr can't be tuned to understand .svnignore files? Is there such a thing? I think you mean the svn:ignore metadata, which I doubt you can get to through either mirror. Even if you could, svn:ignore is pretty weak (only applies to the directory it is set on and can only do glob patterns). .hgignore is significantly more robust (and only has to be written to a single spot) since matches entire paths and supports regex. Eventually, we'll need to build an appropriate .hgignore file for the python tree anyways. As with others, I don't see the harm in committers who use those tools adding and maintaining these files. Seems akin to having Misc/python-mode.el and Misc/Vim/python.vim. It's all in the spirit of supporting the tools that people are actually using. -- Scott Dial [email protected] [email protected] -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85934 - in python/branches/py3k: Misc/NEWS Modules/socketmodule.c
On 10/30/2010 4:08 PM, Martin v. Löwis wrote: >> I think size should be in TCHARs, not in bytes. (MSDN says so) >> And GetComputerName's signature differs from MSDN. (Maybe should >> use GetComputerNameExW again?) > > You are right. So how about this patch? Still not quite right. The call to GetComputerNameExW after ERROR_MORE_DATA (which gives the number of *bytes* needed) still needs to pass "size/sizeof(wchar_t)" back into GetComputerNameExW since it wants the number TCHARs. I don't think the +1 is needed either (MSDN says it already included the null-terminator in the byte count. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 11/7/2010 7:09 PM, Martin v. Löwis wrote: >> Luckily, the problems that we faced 2.5 years ago when I came up with >> the idea of Snakebite are still just as ever present today ;-) > > Is this bashing of existing infrastructure really necessary? > People (like me) might start bashing about vaporware and how > a bird in the hand is worth two in the bush. Cooperate, don't > confront. +1 Respect your (software) elders. The Snaketbite rhetoric has always been less than generous with regard to Buildbot, but Buildbot has been providing an infinitely more useful service to the community for much longer than Snakebite has for those 2.5 years. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 11/7/2010 9:58 PM, C. Titus Brown wrote: > Yes, yes, I agree that some graciousness is a good idea. > > Oh, wait... you're not helping. Classy. I don't remember being invited to help. snakebite.org is a dead end. snakebite-list hasn't had a post for over a year. Where is the list of things that you need done so that I can get started on that? Oh wait.. Seriously, all I asked was for you to tone down your insults to a technology that is already solving problems today. Why you feel the need to attack me personally is beyond my understanding. Furthermore, I don't see why I need to be "helping" -- somebody who doesn't want help -- to be able to deduce that your message is being insulting to the authors of Buildbot. On 11/7/2010 9:58 PM, C. Titus Brown wrote: > And I'm not in a personal position to help, so I've basically tried > to shut up about it :). At least I am in good company. ;) -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "Missing" 2.5 feature
Tim Peters wrote: > Just to make life harder ;-), I should note that code, docs and tests > for sys._current_frames() are done, on the tim-current_frames branch. > All tests pass, and there are no leaks in the new code. It's just a > NEWS blurb away from being just another hectic release memory :-) Wouldn't this function be better named sys._getframes since we already have a sys._getframe for getting the current frame? -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capabilities / Restricted Execution
Phillip J. Eby wrote: > A function's func_closure contains cell objects that hold the > variables. These are readable if you can set the func_closure of some > function of your own. If the overall plan includes the ability to restrict > func_closure setting (or reading) in a restricted interpreter, then you > might be okay. Except this function (__getattribute__) has been trapped inside of a class which does not expose it as an attribute. So, you shouldn't be able to get to the func_closure attribute of the __getattribute__ function for an instance of the Guard class. I can't come up with a way to defeat this protection, at least. If you have a way, then I'd be interested to hear it. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Capabilities / Restricted Execution
Talin wrote: > Scott Dial wrote: >> Phillip J. Eby wrote: >> >>> A function's func_closure contains cell objects that hold the >>> variables. These are readable if you can set the func_closure of some >>> function of your own. If the overall plan includes the ability to restrict >>> func_closure setting (or reading) in a restricted interpreter, then you >>> might be okay. >> >> Except this function (__getattribute__) has been trapped inside of a >> class which does not expose it as an attribute. So, you shouldn't be >> able to get to the func_closure attribute of the __getattribute__ >> function for an instance of the Guard class. I can't come up with a way >> to defeat this protection, at least. If you have a way, then I'd be >> interested to hear it. > > I've thought of several ways to break it already. Some are repairable, > I'm not sure that they all are. > > For example, neither of the following statements blows up: > > print t2.get_name.func_closure[0] > print object.__getattribute__( t2, '__dict__' ) > > Still, its perhaps a useful basis for experimentation. > > -- Talin I quickly poked around it in python and realized that in 2.5 (as opposed to the 2.4 python I was playing in) the cell object exposes cell_contents.. blargh. So, yes, you can defeat the protection because the wrapped instance is exposed. print t2.get_name() t2.get_name.func_closure[0].cell_contents.im_self.name = 'poop' print t2.get_name() Although, your second example with using the object.__getattribute__ doesn't seem to really be an issue. You retrieved the __dict__ for the Guard class which is empty and is something we should not feel concerned about being leaked. Only way I see this as viable is if in "restricted" mode cell_contents was removed from cell objects. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with super() usage
Greg Ewing wrote:
> Guido van Rossum wrote:
>> In the world where cooperative multiple inheritance
>> originated (C++), this would be a static error.
>
> I wasn't aware that C++ had anything resembling super().
> Is it a recent addition to the language?
>
It is much more explicit, but you call the function from the
superclass's namespace:
class B : public A {
public:
void m(void) {
A::m(); // The call to my super
};
};
C++ has no concept of MRO, so "super()" would be completely ambiguous.
In fact, if you try to replicate your code in C++ using a generic M
class (which defines a dummy m method), you'll get such an error from
your compiler: "`M' is an ambiguous base of `C'"
This is very much a dynamic language quirk that you can call out to a
function that may or may not exist, and we should avoid comparing it to
other languages which don't allow it. I agree with Guido that in python,
the reasonable fix is to have a superclass which defines an empty method.
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Document performance requirements?
Between the two of you, I think you have made the case that the language specification is better to not include such details. As you both note, it is difficult to capture the essence of what is desired from the performance of the implementation. To tag on other version, what about Big-O space concerns with things like list.sort. I'm sure there are other things to add as well. It seems reasonable to me that everyone has the same interests in mind when they write a program. Make it good, make it fast, make it small, etc. These sort of details should work themselves out if they are actually important. All of these algorithms should be treated as implementation accidents. Having the information about CPython's implementation in the docs would be good. And go most of the way towards having everyone on the same page. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] dict containment annoyance
Jean-Paul Calderone wrote: > def blacklisted(o): > try: > # Is the object contained in the blacklist set? > return o in _blacklistset > except TypeError: > # If it *cannot* be contained in the blacklist set, > # then it probably isn't. > return False I feel confident that the parent could've wrote this snippet himself. His point was to discuss the semantic logic of "x in y" throwing and exception when x can't be in y (because of the TypeError you are catching in your snippet). FWIW, I think the logic of swallowing the TypeError is completely reasonable. It is surprising in a way that you are presented with an exception. However, I can't say I have ever ran into this. For my money, it would make the most sense to have your own blacklistset type. class BlacklistSet(dict): def __contains__(self, x): try: hash(x) except TypeError: return False return super(BlacklistSet, self).__contains__(x) It's however unfortunate that you cannot use the handy curly-braces anymore. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] NotHashableError? (Re: dict containment annoyance)
Greg Ewing wrote: > Guido van Rossum wrote: >> try: >> hash(x) >> except TypeError: >> # apparently x is not hashable >> >> then you're also swallowing any type errors in the computation of a >> legitimate hash function. > > Maybe it would help if there were a specific exception, > such as NotHashableError, that hash functions were > expected to raise in this situation instead of a > generic TypeError. I was going to suggest this, but then I realized that what is really desirable is a check for hashability. But since hashability is a deep property, it is difficult to check for.. seems like an very specific exception type would be a simple solution. FYI, I appreciate that swallowing all TypeErrors is bad form, so it would be nice if it was possible to differentiate ;-) Some brief googling on the subject yield a discussion on py3k about this very subject. http://mail.python.org/pipermail/python-3000/2006-July/002697.html Guido wrote: > Personally, I'm not sure this problem needs solving; I don't recall > ever needing to know whether something is hashable. So perhaps it's of > purely theoretical importance? That would suit me fine given the above > dilemma... > > --Guido -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.spawnlp() missing on Windows in 2.4?
Guido van Rossum wrote: > I just got a report from a Windows user that os.spawnlp() is missing > from Python 2.4, despite being mentioned in the docs. Can someone > confirm this? My Windows box is resting. :-) > "Availability: Unix, Windows. spawnlp(), spawnlpe(), spawnvp() and spawnvpe() are not available on Windows. New in version 1.6." One could argue that it presented poorly, but it reads completely correct. Alternatively one could says "The 'p' variants are unavailable on Windows." but that would be assuming someone understand there was a scheme to the names :-) -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Guido van Rossum wrote: > On 9/21/06, Greg Ewing <[EMAIL PROTECTED]> wrote: >> I think it goes further than that -- each module should >> (potentially) have its own unique view of the module >> namespace, defined at the time the module is installed, >> that can't be disturbed by anything that any other module >> does. > > Well, maybe. But there's also the requirement that if packages A and B > both import C, they should get the same C. Having multiple versions of > the same package loaded simultaneously sounds like a recipe for > disaster. I have exactly this scenario in my current codebase for a server. It was absolutely necessary for me to update a module in Twisted because all other solutions I could come up with were less desirable. Either I send my patch upstream and wait (can't wait), or I fork out another version and place it at the top of sys.path (this seems ok). Except an even better solution is to maintain my own subset of Twisted, because I am localized to a particularly small corner of the codebase. I can continue to use upstream updates to the rest of Twisted without any fussing about merging changes and so forth. And if Twisted was allowed to decide how it saw its own world, then I would have to go back to maintaining my own complete branch. While I don't strictly need to be able to do this, I wanted to at least raise my hand and say, "I abuse this facet of the current import mechanism." -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BUG (urllib2) Authentication request header is broken on long usernames and passwords
The Doctor What wrote:
> The problem is that base64.encodestring() adds newlines to wrap the
> encoded characters at the 76th column.
>
The encodestring is following RFC 1521 which speficies:
The output stream (encoded bytes) must be represented in lines of no
more than 76 characters each. All line breaks or other characters
not found in Table 1 must be ignored by decoding software.
In retrospect, perhaps "{de|en}codestring" was a poor name choice.
urllib2 should be calling b64encode directly.
I have submitted a patch to the tracker: [ 1574068 ] urllib2 - Fix line
breaks in authorization headers.
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PATCH submitted: Speed up + for string Re: PATCHsubmitted: Speed up + for string concatenation, now as fast as "".join(x) idiom
Chetan Pandya wrote:
> My statement wasn't clear enough.
>
> Rendering occurs if the string being concatenated is already a
> concatenation object created by an earlier assignment.
>
I'm not sure how you came to that conclusion. My reading of the patch
doesn't suggest that at all. The operation of string_concat would not
produce what you suggest. I can't find anything anywhere that would
cause the behavior you suggest. The only case where this is true is if
the depth of the tree is too great.
To revisit your example, I will notate concats as string pairs:
a = "Value a =" # new string
a += "anything" # ("Value a =", "anything")
c = a + b# (("Value a =", "anything"), b)
c += "Something" # ((("Value a =", "anything"), b), "Something"
So again, for your other example of repeated right-hand concatenation,
you do not continually render the concat object, you merely create new
one and attach to the leaves. Once the print is executed, you will force
the rendering of the object, but only once that happens.
So in contrast to your statement, there are actually there are fewer
allocations of strings and smaller objects being allocated than the
current trunk uses.
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 355 status
[EMAIL PROTECTED] wrote: > Talin writes: > > (one additional postscript - One thing I would be interested in is an > > approach that unifies file paths and URLs so that there is a consistent > > locator scheme for any resource, whether they be in a filesystem, on a > > web server, or stored in a zip file.) > > +1 > > But doesn't file:/// do that for files, and couldn't we do something > like zipfile:///nantoka.zip#foo/bar/baz.txt? Of course, we'd want to > do ziphttp://your.server.net/kantoka.zip#foo/bar/baz.txt, too. That > way leads to madness > It would make more sense to register protocol handlers to this magical unification of resource manipulation. But allow me to perform my first channeling of Guido.. YAGNI. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP: Adding data-type objects to Python
Travis Oliphant wrote: > Paul Moore wrote: >> Enough of the abstract. As a concrete example, suppose I have a (byte) >> string in my program containing some binary data - an ID3 header, or a >> TCP packet, or whatever. It doesn't really matter. Does your proposal >> offer anything to me in how I might manipulate that data (assuming I'm >> not using NumPy)? (I'm not insisting that it should, I'm just trying >> to understand the scope of the PEP). >> > > What do you mean by "manipulate the data." The proposal for a > data-format object would help you describe that data in a standard way > and therefore share that data between several library that would be able > to understand the data (because they all use and/or understand the > default Python way to handle data-formats). > Perhaps the most relevant thing to pull from this conversation is back to what Martin has asked about before: "flexible array members". A TCP packet has no defined length (there isn't even a header field in the packet for this, so in fairness we can talk about IP packets which do). There is no way for me to describe this with the pre-PEP data-formats. I feel like it is misleading of you to say "it's up to the package to do manipulations," because you glanced over the fact that you can't even describe this type of data. ISTM, that you're only interested in describing repetitious fixed-structure arrays. If we are going to have a "default Python way to handle data-formats", then don't you feel like this falls short of the mark? I fear that you speak about this in too grandiose terms and are now trapped by people asking, "well, can I do this?" I think for a lot of folks the answer is: "nope." With respect to the network packets, this PEP doesn't do anything to fix the communication barrier. Is this not in the scope of "a consistent and standard way to discuss the format of binary data" (which is what your PEP's abstract sets out as the task)? -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] MSI being downloaded 10x more than all other files?!
Guido van Rossum wrote: > What could cause this dramatic popularity of Python on Windows? You should ask yourself: 1) Where else can people grab Python on Windows? 2) Where else can people grab Python for [every other operating system]? Most distros are kind enough to provide their own mirror, I would say that easily accounts for the discrepancy since I am not aware of any other place to grab a windows install. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
Martin v. Löwis wrote:
> It would still "suffer" from the cross-module issue:
>
> # a.py
> from __future__ import items_is_iterator
> def f(d):
> return d.items
> # b.py
> import a
> d = {}
> print a.f(d)
>
> For compatibility with 2.x, a.f should really return a bound
> method that returns lists; for compatibility with 3.x, it will return
> a method that produces an iterator.
>
> Of course, one might say "don't do that, then".
>
This seems like a perfectly reasonable thing to say. If I am writing
something for 2.X and want to use a 3.X feature switch, then I am
responsible for maintaining the external API, be it the 2.X version of
d.items if that is what is expected by the user.
Your complaint only works if a.f was pre-defined to return a list, which
the developer of "a" would know about when he decided to flip on
items_is_iterator. As a part of that refactoring, he would patch up the
return to be list(d.items) to maintain the API he has already defined.
It is not clear by your example that "b" wasn't in fact expecting to get
an iterator, which is now a great deal simpler thanks to the future
statement.
My point is simply that this argument doesn't work unless you assume the
developer of "a" has failed to do his job (which is to maintain the
external API by also testing the external API).
BTW, I patched your code because both versions you have presented didn't
actually work.
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Brett Cannon wrote: > On 2/12/07, Guido van Rossum <[EMAIL PROTECTED]> wrote: >> I recommend that you do some experiments with the readability of the >> .[...] notation, e.g. write a program that randomly generates x.[foo] >> and x[foo], and see how fast you can spot the difference. I bet that >> you won't have any trouble. >> > > OK, so real-world examples. First, ``foo.(name)``, from urllib.py:: > [snip] > > Neither version jumps out at me strongly, although between the two the > ``.[]`` version shows up the best. But that might also be because of > the lower noise when used in a call. > > -Brett After seeing a real-world example of the notation, I'm not a fan. Like Brett, of the two I find ".[]" to be the most readable. I do in fact immediately associate an identifier next to "()" with being a function call, regardless of the dot. The "[]" immediately associates me with a type of dereferencing action which is more appropriate for the context. The danger here is that "foo.bar" and "baz.(bar)" use "bar" in drastically different ways and it is not made obvious enough. "baz.[bar]" gets you closer because we are used to seeing "bar" as a variable containing a key (as opposed to "foo.bar" where 'bar' is in fact the key). At the risk of needing flame retardant, I would propose the use of something more visually jarring. The first thing that came to mind was something like "[EMAIL PROTECTED]". The only usage of the @ symbol I can recall is in Ruby where it is used instead of "self." for attribute access, which is in the right ballpark for what we mean here and the brackets differentiate it still. To borrow the urllib2.py example: if attr[:12] == '_Request__r_': name = attr[12:] if hasattr(Request, 'get_' + name): [EMAIL PROTECTED]'get_' + name]() return [EMAIL PROTECTED] raise AttributeError, attr -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
Raymond Hettinger wrote: > Rather than munge existing syntaxes, an altogether new one would be more > clear: > >self->name = self.metadata->name > My problem with this is that it isn't a "name". It should grammatically be a test (i.e. it can take on the expression any function argument could take). How do you spell "getattr(self, 'req_' + state)" with that arrow? You need some kind of grouping delimiters to make this operator be a syntax benefit otherwise you didn't shorten anything. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encouraging developers
As an outsider who has submitted a handful of patches and has always wanted to become more involved.. I would like to comment as I feel like I am the target audience in question. I apologize ahead of time if I am speaking out of place. Martin v. Löwis wrote: > Phil Thompson schrieb: >> 1. Don't suggest to people that, in order to get their patch reviewed, they >> should review other patches. The level of knowledge required to put together >> a patch is much less than that required to know if a patch is the right one. > > People don't *have* to review patches. They just can do that if they > want expedite review of their code. > While I understand that this tit-for-tat mechanism is meant to ensure participation, I believe in reality it doesn't, as the 400-some outstanding patches you referenced elswhere indicate. I can personally attest to having a patch that is over a year old with no "core developer" having any interest at all with the subject matter. And to be frank, nor did I really, but I saw a problem and was capable of solving it. My lack of caring about the patch means I am not going to beat people over the head to pay attention. This system is broken for someone like me (coder) that just wants to help out (non-coders). >> 2. Publically identify the core developers and their areas of expertise and >> responsibility (ie. which parts of the source tree they "own"). > > I doubt this will help. Much of the code isn't owned by anybody > specifically. Those parts that are owned typically find their patches > reviewed and committed quickly (e.g. the tar file module, maintained by > Lars Gustäbel). If nothing else, as an outsider there is no way to know why your patch gets ignored while others get swiftly dealt with. Any sort of information like this would at least provide more transparency in what may appear to be elitest processes. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encouraging developers
Martin v. Löwis wrote: > Paul Moore schrieb: >> Here's a random offer - let me know the patch number for your patch, >> and I'll review it. > > Surprisingly (and I hope Scott can clarify that), I can't find anything. > Assuming Scott's SF account is "geekmug", I don't see any open patches > (1574068 was closed within 20 days by amk, last October). > Sadly the sf tracker doesn't let you search for "With comments by". The patch I was making reference to was 1410680. Someone else actually had wrote a patch that contained bugs and I corrected them. And with that, I was the last person to comment or review the patch in question. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale for NamedTemporaryFile?
Greg Ewing wrote: > Gregory P. Smith wrote: >> I prefer the default of "clean up after itself" as is to avoid leaving >> temporary file turds on systems caused by us lazy programmers. > > Maybe instead of an option there should be a separate > function with a different name, such as NewUniqueFile. > For the use cases I have in mind, the file isn't really > "temporary" at all. Or rather, only the name is temporary, > as it ultimately gets renamed. > I'm in favor of adding such a function. I've already wrote my way around this missing feature before. The tempfile modules AFAIK is the only portable way to make a unique filename and open it without exposing a race condition. As it is, it's not that difficult to write this function yourselves using mkstemp directly, but I believe there is a great benefit to have it in the stdlib. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale for NamedTemporaryFile?
Greg Ewing wrote: > A tangential question -- why are TemporaryFile and > NamedTemporaryFile named in TitleCase, when they're > functions and not classes? > I wondered the same. At first draft of my email I wrote "class" operating under the assumption that only classes got to be camel-cased. If I had to guess, the rationale was that they are simply wrappers a class. Nevertheless, tempfile dates well back before the PEP 8 style guidelines. I think consistency would dictate that a newly added function should match the other ones, but I have no strong feelings about this. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rationale for NamedTemporaryFile?
Greg Ewing wrote: > A tangential question -- why are TemporaryFile and > NamedTemporaryFile named in TitleCase, when they're > functions and not classes? > I wondered the same. At first draft of my email I wrote "class" operating under the assumption that only classes got to be camel-cased. If I had to guess, the rationale was that they are simply wrappers a class. Nevertheless, tempfile dates well back before the PEP 8 style guidelines. I think consistency would dictate that a newly added function should match the other ones, but I have no strong feelings about this. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] possible urllib bug on Windows XP
Shane Geiger wrote: > Is this possibly a bug on Windows XP? I can't reproduce this bug on Windows XP. -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.rename on windows
Raghuram Devarakonda wrote: > As a last resort, I > checked cygwin documentation which claims that it's rename() is > POSIX.1 compliant. If I am not mistaken, POSIX.1 does require > atomicity so I am curious how rename() is implemented there. The cygwin implementation of rename goes like this: 1) Try to use MoveFile 2) Try to use MoveFileEx(..., MOVEFILE_REPLACE_EXISTING) 3) Try to unlink destination, then try to use MoveFile And as you say, Cygwin claims it meets POSIX.1. And, POSIX.1 says, "If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing." Clearly, unliking and then calling MoveFile is not atomic. So, cygwin is not being honest here because in these less frequent cases, the rename will not be atomic. Also note, MVCRT only tries step 1 of cygwin's version. Which I believe also suggests that it's the only version that is atomic. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Summary of Tracker Issues
Hrvoje Niksic wrote: > On Wed, 2007-05-16 at 22:17 -0700, Talin wrote: >> Here's a simple method: Put up a free porn site [...] > > Is it known that someone actually implemented this? I moderate a discussion forum which was abused with this exact attack. At the time, it was a phpBB forum which had the standard graphical captcha. After switching to a different forum package, the attacks went away. I will assume because (as it has been said) it was no longer a well-known and common interface. However, it may also be because instead of using a graphic (which is easily transplanted to another page), it uses ascii art which would require more effort to extract and move to another page. -Scott -- Scott Dial [EMAIL PROTECTED] [EMAIL PROTECTED] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] futures API
On 12/11/2010 9:44 AM, Thomas Nagy wrote: > The amount of work items processed by unit of time does not seem to be a > straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out > of curiosity, what is the "_thread_references" for? > > The source file for the example is in: > http://www.freehackers.org/~tnagy/futures_test3.py > > The diagram was created by: > http://www.freehackers.org/~tnagy/futures_test3.plot You're test code does 50,000,000 of list appends. I suspect your benchmark is telling you more about the behavior of large lists than the overhead of the futures module. You should retry that experiment with the list pre-allocated. Beyond that, the curve in that line is not exactly a large amount of variance from a straight line. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable
On 12/18/2010 8:50 AM, James Y Knight wrote: > I think instead of calling abort() to kill the process, you should: > - install the signal handler with SA_NODEFER|SA_RESETHAND (or if > sigaction is not present, explicitly reset the action to SIG_DFL and > unblock first thing upon entering the handler), and then, > - at the end of the handler, kill(getpid(), orig_signal) in order to > abort the process. I concur with this being the correct way to right such a handler. > This has two advantages: 1) the process's exit code will actually > show the correct signal, It's more than an advantage: it's the only correct way to handle a termination signal. > 2) it might let the OS fault handlers work properly > as well -- I'm not sure. If it does, you may want to experiment with > whether having or omitting SA_NODEFER gives a better backtrace (from the > OS mechanism) in that case. Even if that doesn't work, things like the grsecurity patches to linux use these signals to detect exploits and log them and do throttling. Calling abort() effectively converts all of these faults into SIGABRT terminations that are considered (more?) innocent terminations. -- Scott Dial [email protected] [email protected] -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Search-friendly shortcuts for Windows?
On 12/20/2010 8:38 PM, Nick Coghlan wrote: > would it be reasonable to consider including the major version number > in the start menu shortcut names? +1 First thing I did was add an x.y prefix to the Python shortcuts. There are a lot of application shortcuts on my Win7 system that have version numbers that have no naming conflicts, so even if a single version of Python was installed, it would not look out of place at all. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exception __name__ missing?
On 1/17/2011 3:22 PM, Ron Adam wrote: > Is this on purpose? This reminds me of something I ran into a few years ago wrt. the attribute on exceptions. Namely, that instances of built-in exceptions do not have a __module__ attribute, but instance of user exceptions do -- a change which appeared in Python 2.5: http://mail.python.org/pipermail/python-list/2007-November/1088229.html I had a use case, using ZSI to provide a SOAP interface, where being able to get the __module__ and __name__ was needed (to serialize into a SOAP "fault" message). I worked around the issue by referencing the __class__ (as the other replier mentioned). But, I didn't receive any responses then, so I think not a lot of attention was put into these type of attributes on exceptions. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
On 1/31/2011 1:38 PM, Brett Cannon wrote: > I should mention that I have considered implementing a caching finder > and loader for filesystems in importlib for people to optionally > install to use for themselves. The real trick, though, is should it > only cache hits, misses, or both? Regardless, though, it would be a > very simple mixin or subclass to implement if there is demand for this > sort of thing. I have in the past implemented a PEP302 finder/loader zipfile-based cache. On campus, I use a version of python installed to my home directory that is on an NFS share. I found such a cache often gave slower startup times for applications like bzr and hg. My cache merely stores things it finds things in sys.path and loads from the zipfile names that it knows and storing those that it doesn't. I make no attempt to invalidate the cache contents once stored. So, I am already talking about a best-case scenario for caching. I'm not sure how you could invalidate the cache without paying the cost of all the normal syscalls that we are trying to avoid. My finder/loader is not bug-free, but I'd be glad to make it available to someone if they want to play around with it. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On 2/23/2011 9:19 PM, Alexander Belopolsky wrote: > On Wed, Feb 23, 2011 at 8:48 PM, Dj Gilcrease wrote: >> Google Code search limited to python >> >> latin1: 3,489 >> http://www.google.com/codesearch?hl=en&lr=&q=latin1+lang%3Apython&sbtn=Search >> latin-1: 5,604 >> http://www.google.com/codesearch?hl=en&lr=&q=latin-1+lang%3Apython&sbtn=Search latin1: 1,618 http://www.google.com/codesearch?hl=en&lr=&q=%28\%22latin1\%22|\%27latin1\%27%29+lang%3Apython&sbtn=Search latin-1: 2,241 http://www.google.com/codesearch?hl=en&lr=&q=%28\%22latin-1\%22|\%27latin-1\%27%29+lang%3Apython&sbtn=Search >> utf8: 25,341 >> http://www.google.com/codesearch?hl=en&lr=&q=utf8+lang%3Apython&sbtn=Search >> utf-8: 179,806 >> http://www.google.com/codesearch?hl=en&lr=&q=utf-8+lang%3Apython&sbtn=Search utf8 9,676 http://www.google.com/codesearch?hl=en&lr=&q=%28\%22utf8\%22|\%27utf8\%27%29+lang%3Apython&sbtn=Search utf-8 44,795 http://www.google.com/codesearch?hl=en&lr=&q=%28\%22utf-8\%22|\%27utf-8\%27%29+lang%3Apython&sbtn=Search > Your search is invalid. You hit things such as Latin1ClassModel which > have no relevance to the issue at hand. You get about the same ratio if you filter out only the quoted strings. -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
