Re: [Python-Dev] Request for comments: [issue22941] IPv4Interface arithmetic changes subnet mask
On 2015-03-12 10:46 AM, Eric V. Smith wrote:
On 3/12/2015 10:00 AM, Antoine Pitrou wrote:
Related issue:
ipaddress.IPv4Interface('10.0.0.255/8') + 1
IPv4Interface('10.0.1.0/32')
Should the result be IPv4Interface('10.0.0.0/8')
(i.e. wrap around)? Should OverflowError be raised?
I think it should be an OverflowError. 10.0.1.0/8 makes no sense, and
10.0.0.0/8 is unlikely to be desirable.
No, you are both imbuing meaning to the dot-notation that is not there.
A.B.C.D is just an ugly way to represent a single 32-bit unsigned
integer. The only contentious part would be when your addition would
overlfow the host-portion of the address:
ipaddress.IPv4Interface('10.255.255.255/8') + 1
IPv4Interface('11.0.0.0/8')
However, if you take the perspective that the address is just a 32-bit
unsigned integer, then it makes perfect sense. I would argue it's likely
to be a source of bugs, but I can't say either way because I never
adopted using this library due to issues that are cataloged in the
mailing list archives. (In the end, I wrote my own and have never found
the need to support such operands.)
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On 2015-03-06 11:34 AM, Brett Cannon wrote: This PEP proposes eliminating the concept of PYO files from Python. To continue the support of the separation of bytecode files based on their optimization level, this PEP proposes extending the PYC file name to include the optimization level in bytecode repository directory (i.e., the ``__pycache__`` directory). As a packager, this PEP is a bit silent on it's expectations about what will happen with (for instance) Debian and Fedora packages for Python. My familiarity is with Fedora, and on that platform, we ship .pyc and .pyo files (using -O for the .pyo). Is it your expectation that such platforms will still distribute -O only? Or also -OO? In my world, all of the __pycache__ directories are owned by root. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enable Hostname and Certificate Chain Validation
On 2014-01-22 9:33 AM, Donald Stufft wrote: For everything but pip, you’d add it to your OS cert store. Pip doesn’t use that so you’d have to use the —cert config. What if I don't want that self-signed cert to be trusted by all users on the system? What if I don't have administrative rights? How do I do it then? Is this common knowledge for average users? Are we trading one big red box in the documentation for another? Anecdotally, I already know of a system at work that is using HTTPS purely for encryption, because the authentication is done in-band. So, a self-signed cert was wholly sufficient. The management tools use a RESTful interface over HTTPS for control, but you are telling me this will be broken by default now. What do I tell our developers (who often adopt the latest and greatest versions of things to play with)? -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460: allowing %d and %f and mojibake
On 2014-01-11 22:09, Nick Coghlan wrote:
For Python 2 folks trying to grok where the bright line is in terms of
the Python 3 text model: if your proposal includes *any* kind of
implicit serialisation of non binary data to binary, it is going to be
rejected as an addition to the core bytes type. If it avoids crossing
that line (as the buffer-API-only version of PEP 460 does), then we can
talk.
To take such a hard-line stance, I would expect you to author a PEP to
strip the ASCII conveniences from the bytes and bytearray types.
Otherwise, I find it a bit schizophrenic to argue that methods like
lower, upper, title, and etc. don't implicitly assume encoding:
a = scott.encode('utf-16')
b = a.title()
c = b.decode('utf-16')
'SCOTT'
So, clearly title() not only depends on the bytes characters encoded in
a superset of ASCII characters, it depends on the bytes being a sequence
of ASCII characters, which looks an awful lot like an operation on an
implicit encoded string.
b文字化け
File stdin, line 1
SyntaxError: bytes can only contain ASCII literal characters.
There is an implicit serialization right there. My terminal is utf8 (or
even if my source encoding is utf8), so why would that not be:
b'\xe6\x96\x87\xe5\xad\x97\xe5\x8c\x96\xe3\x81\x91'
I sympathize with Ethan that the bytes and bytearray types already seem
to concede that bytes is the type you want to use for 7-bit ASCII
manipulations. If that is not what we want, then we are not doing a good
job communicating that to developers with the API. At the onset, the
bytes literal itself seems to be an attractive nuisance as it gives a
nod to using bytes for ASCII character sequences (a.k.a ASCII strings).
Regards,
-Scott
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.x vs 3.x survey results
On 2014-01-02 17:54, Dan Stromberg wrote: I put it at https://wiki.python.org/moin/2.x-vs-3.x-survey It would've been nice to see some crosstabs. Pretty much any question after Q3 is incomprehensible without splitting the respondents into sub-groups first. Of the 2.49% of people who said they've never written Python 2.x, how many of those people also said they have never written Python 3.x too? (There really is 4 categories of developers being surveyed here.) Of the 22.91% of people who said Python 3.x was a mistake, how many of them also said they have never written any Python 3.x? Of the 40% of people who said they have never written Python 3.x, how many of them also said they had dependencies keeping them on Python 2.x? Etc. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] please back out changeset f903cf864191 before alpha-2
On 8/26/2013 8:51 AM, Antoine Pitrou wrote: Le Mon, 26 Aug 2013 08:24:58 -0400, Tres Seaver tsea...@palladion.com a écrit : On 08/26/2013 04:36 AM, Antoine Pitrou wrote: event-driven processing using network libraries Maybe I missed something: why should considerations from that topic influence the design of an API for XML processing? Because this API is mostly useful when the data is received (*) at a slow enough speed - which usually means from the network, not from a hard drive. ... The whole *point* of adding IncrementalParser was to parse incoming XML data in a way that is friendly with event-driven network programming, other use cases being *already* covered by existing APIs. This is why it's far from nonsensical to re-use an existing terminology from that world. Since when is Tulip the OOWTDI? If this was Twisted, it would be write and finish[1]. Tulip's Protocol ABC isn't even a good match for the application. There is reason that Twisted has a separate Consumer/Producer interface from the network I/O interface. I'm sure there is other existing practice in this specific area too (e.g., XMLParser). [1] http://twistedmatrix.com/documents/13.1.0/api/twisted.protocols.ftp.IFinishableConsumer.html -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 6/23/2013 8:16 PM, Raymond Hettinger wrote: Yes, this is a micro-optimization. In working on implementing deque slicing for 3.4, I restudied the block access patterns. On an appendleft(), popleft() or extendleft() operation, the left link is accessed immediately before or after the leftmost entry in the data block. The old struct arrangement can cause an unnecessary cache miss when jumping leftward. This was something I overlooked when I originally wrote the code almost a decade ago. A decade ago, cache lines were 64 bytes, pointers were 4 bytes, and allocations were 16 byte aligned, so there could never be a cache miss. Nowadays, cache lines are still 64 bytes but pointers are 8 bytes, and we still allocating on 16 byte alignment, so you have a 25% chance of a cache miss now. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod
On 6/22/2013 2:17 PM, Benjamin Peterson wrote:
Many people have raised concerns about this change, so I've now backed it out.
I think that change also goes with this change:
http://hg.python.org/cpython/rev/f1dc30a1be72
changeset 84248:f1dc30a1be72 2.7
Arrange structure to match the common access patterns.
1.1 --- a/Modules/_collectionsmodule.c
1.2 +++ b/Modules/_collectionsmodule.c
1.3 @@ -48,8 +48,8 @@
1.4
1.5 typedef struct BLOCK {
1.6 struct BLOCK *leftlink;
1.7 +PyObject *data[BLOCKLEN];
1.8 struct BLOCK *rightlink;
1.9 -PyObject *data[BLOCKLEN];
1.10 } block;
1.11
1.12 #define MAXFREEBLOCKS 10
, which seems like a strange micro-optimization, at best.
Based on that structure, it would seem that neither BLOCKLEN being 62
(previous value) nor 64 (the new value) make much sense. It would seem
best that sizeof(block) == 64, so BLOCKLEN should be (64 -
2*sizeof(PyObject *)). Nevertheless, I am skeptical that any tuning of
this structure provides any meaningful performance improvement.
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 6/18/2013 4:40 PM, Victor Stinner wrote:
No context argument
---
Simplify the signature of allocator functions, remove the context
argument:
* ``void* malloc(size_t size)``
* ``void* realloc(void *ptr, size_t new_size)``
* ``void free(void *ptr)``
It is likely for an allocator hook to be reused for
``PyMem_SetAllocator()`` and ``PyObject_SetAllocator()``, or even
``PyMem_SetRawAllocator()``, but the hook must call a different function
depending on the allocator. The context is a convenient way to reuse the
same custom allocator or hook for different Python allocators.
I think there is a lack of justification for the extra argument, and the
extra argument is not free. The typical use-case for doing this
continuation-passing style is when the set of contexts is either
unknown, arbitrarily large, or infinite. In other words, when it would
be either impossible or impractical to enumerate all of the contexts.
However, in this case, we have only 3.
Your proposal already puts forward having 3 pairs of Get/Set functions,
so there is no distinct advantage in having a single typedef instance
that you pass in to all 3 of them. And, having all 3 pairs use the same
typedef is a bit of an attractive nuisance, in that one could pass the
wrong allocators to the wrong setter. With that, I could argue that
there should be 3 typedefs to prevent coding errors.
Nevertheless, the ctx argument buys the implementer nothing if they have
to begin their alloc function with if(ctx == X). In other words, there
is nothing simpler about:
void *_alloc(void *ctx, size_t size) {
if(ctx == PYALLOC_PYMEM)
return _alloc_pymem(size);
else if(ctx == PYALLOC_PYMEM_RAW)
return _alloc_pymem_raw(size);
else if(ctx == PYALLOC_PYOBJECT)
return _alloc_pyobject(size);
else
abort();
}
PyMemBlockAllocator pymem_allocator =
{.ctx=PYALLOC_PYMEM, .alloc=_alloc, .free=_free};
PyMemBlockAllocator pymem_raw_allocator =
{.ctx=PYALLOC_PYMEM_RAW, .alloc=_alloc, .free=_free};
PyMemBlockAllocator pyobject_allocator =
{.ctx=PYALLOC_PYOBJECT, .alloc=_alloc, .free=_free};
In comparison to:
PyMemBlockAllocator pymem_allocator =
{.alloc=_alloc_pymem, .free=_free_pymem};
PyMemBlockAllocator pymem_raw_allocator =
{.alloc=_alloc_pymem_raw, .free=_free_pymem};
PyMemBlockAllocator pyobject_allocator =
{.alloc=_alloc_pyobject, .free=_free_pyobject};
And in the latter case, there is no extra indirect branching in the
hot-path of the allocators.
Also, none of the external libraries cited introduce this CPS/ctx stuff.
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators
On 6/18/2013 11:32 PM, Nick Coghlan wrote: Agreed more of that rationale needs to be moved from the issue tracker into the PEP, though. Thanks for the clarification. I hadn't read the issue tracker at all. On it's face value, I didn't see what purpose it served, but having read Kristján's comments on the issue tracker, he would like to store state for the allocators in that ctx pointer.[1] Having read that (previously, I thought the only utility was distinguishing which domain it was -- a small, glorified enumeration), but his use-case makes sense and definitely is informative to have in the PEP, because the utility of that wasn't obvious to me. Thanks, -Scott [1] http://bugs.python.org/issue3329#msg190529 One particular trick we have been using, which might be of interest, is to be able to tag each allocation with a context id. This is then set according to a global sys.memcontext variable, which the program will modify according to what it is doing. This can then be used to track memory usage by different parts of the program. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/23/2013 10:53 AM, R. David Murray wrote: Ah. I'd be looking for a bug every time I saw isinstance(value, myEnumClass). A better class name for values and an API for getting that class from the EnumClass would be nice, though. (So that you could write isinstance(value, MyEnumClass.ValueClass), say.) Reading what you have wrote, it seems that the issue is whether you consider an instance of Enum a thing or a class of things. If you think of it as a thing, then C is a object that has attributes for other things that are not like C. However, if you think of C as a class of things, then C having attributes that are instances of it's type is completely natural. Fundamentally, the question is whether an instance of Enum is a new type or an instance. And for me, it's a new type and I expect enum values to be instance of that type. -Scott -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/23/2013 11:58 AM, Guido van Rossum wrote: You seem to be mixing up classes and metaclasses. I was trying to explain it in more plain terms, but maybe I made it even more confusing.. Anyways, I agree with you that isinstance(Color.RED, Color) should be True. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library
On 4/12/2013 10:51 PM, Steven D'Aprano wrote: And two examples from asm-generic/errno.h: #define EWOULDBLOCK EAGAIN /* Operation would block */ #define EDEADLOCK EDEADLK That's actually even better of an example than you may have realized because historically EWOULDBLOCK != EAGAIN[1]. So, there very well may need to exist such code as: if some hairy platform: _EAGAIN = X _EWOULDBLOCK = Y else: _EAGAIN = _EWOULDBLOCK = X class Errno(Enum): EAGAIN = _EAGAIN EWOULDBLOCK = _EWOULDBLOCK I don't think it's all that uncommon that enum values that represent states of a system get merged or renamed over time, and this one is a great example of that. [1] http://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Error-Codes.html#index-EAGAIN-97 -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] sys.implementation
On 5/10/2012 1:40 AM, Nick Coghlan wrote: Unordered types can be a PITA for testing, for display and for generic serialisation, so I definitely want to see a PEP before we add a new one that basically has its sole reason for existence being you can iterate over and index the field values in a namedtuple. I could use those same arguments (testing, display, and generic serialization) as reasons /against/ using an ordered type (when it's not the intent of the author that it be ordered). That is: - Testing: This is an attractive nuisance because adding fields later can break the tests if the author of the type had no intent on the ordering being guaranteed (or the number of fields). - Display: If the author of the type didn't intend on the ordering being guaranteed, then the display could become nonsense when changing versions (e.g., upgrading a 3rd-party library). - Generic Serialization: Again, if the author didn't plan for that, then they could add additional fields or re-arrange them in a way that makes naive serialization give incorrect instances. The point is that the author of the type can't protect you from these mistakes if a namedtuple is used. The only tool the author of the type has at their disposal to warn you of your ways is documentation. If the type doesn't support iteration or indexing, then you are forced to do it right, because it's the only way that works. Furthermore, what is wrong with a repr that yields a dict-like string record(a=1, b=2, c=3) with regard to testing and display? -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .{git,bzr}ignore in cpython HG repo
On 4/2/2012 11:03 AM, Chris Angelico wrote: On Mon, Apr 2, 2012 at 9:50 PM, Antoine Pitrou solip...@pitrou.net wrote: That said, these files will always be outdated, so we might as well remove them so that at least git / bzr users don't get confused. Apologies for what may be a stupid suggestion, but is it possible to write a script that generates .gitignore and .bzrignore from .hgignore? That ought to solve the problem - take the former two out of the repository, and everyone who wants to use git or bzr can simply generate them on requirement. In general, Hg's ignore files are more expressive (regex and globbing) than Git's ignore files (globbing only). Our .hgignore file has regex rules, but if someone was so inclined, they could expand those rules based on their current HEAD. I do not know if such a tool already exists in the wild. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .{git,bzr}ignore in cpython HG repo
On 3/30/2012 11:00 AM, Matěj Cepl wrote:
It is even worse, that (understandingly) .{bzr,git}ignore are apparently
poorly maintained, so in order to get an equivalent of .hgignore in
.gitignore, one has to apply the attached patch.
Create an issue on the bug tracker. In the meantime, you can either
commit the change to your clone, or you can put your ignores into
.git/info/exclude. No reason to be so sore about it, since Git lets you
have your own ignore file without requiring it be a tracked file.
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On 3/27/2012 8:36 PM, Victor Stinner wrote: Scott wrote: Scott monotonic_clock = always goes forward but can be adjusted Scott steady_clock = always goes forward and cannot be adjusted I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux 2.6.28. I don't know if other clocks used by my time.monotonic() proposition can be adjusted or not. If I understand correctly, time.steady() cannot be implemented using CLOCK_MONOTONIC on Linux because CLOCK_MONOTONIC can be adjusted? Does it really matter if a monotonic speed is adjusted? You are right that CLOCK_MONOTONIC can be adjusted, so the Boost implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right either due to suspend -- there doesn't appear to be a POSIX or Linux clock that is defined that meets the steady definition. I am not familiar enough with Windows or Mac to know for certain whether the Boost implementation has the correct behaviors either. With that in mind, it's certainly better that we just provide time.monotonic() for now. If platform support becomes available, then we can expose that as it becomes available in the future. In other words, at this time, I don't think time.steady() can be implemented faithfully for any platform so lets just not have it at all. In that case, I don't think time.try_monotonic() is really needed because we can emulate time.monotonic() in software if the platform is deficient. I can't imagine a scenario where you would ask for a monotonic clock and would rather have an error than have Python fill in the gap with an emulation. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On 3/28/2012 4:48 AM, Victor Stinner wrote: Scott monotonic_clock = always goes forward but can be adjusted Scott steady_clock = always goes forward and cannot be adjusted I don't know if the monotonic clock should be called time.monotonic() or time.steady(). The clock speed can be adjusted by NTP, at least on Linux 2.6.28. (...) You are right that CLOCK_MONOTONIC can be adjusted, so the Boost implementation is wrong. I'm not sure that CLOCK_MONOTONIC_RAW is right either due to suspend -- there doesn't appear to be a POSIX or Linux clock that is defined that meets the steady definition. The term adjusted should be clarified. A clock can be adjusted by setting its counter (e.g. setting the system date and time) or by changing temporary its frequency (to go faster or slower). Linux only adjusts CLOCK_MONOTONIC frequency but the clock is monotonic because it always goes forward. The monotonic property can be described as: t1=time.monotonic() t2=time.monotonic() assert t2 = t1 I agree. The point I was making is that implication of steady is that (t2-t1) is the same (given that t2 and t1 occur in time at the same relative moments), which is a guarantee that I don't see any platform providing currently. Any clock that can be adjusted in any manner is not going to meet the steady criterion. In that case, I don't think time.try_monotonic() is really needed because we can emulate time.monotonic() in software if the platform is deficient. As I wrote, I don't think that Python should workaround OS bugs. If the OS monotonic clock is not monotonic, the OS should be fixed. I sympathize with this, but if the idea is that the Python stdlib should use time.monotonic() for scheduling, then it needs to always be available. Otherwise, we are not going to use it ourselves, and what sort of example is that to set? I can't imagine a scenario where you would ask for a monotonic clock and would rather have an error than have Python fill in the gap with an emulation. Sorry, I don't understand what you mean with fill in the gap with an emulation. You would like to implement a monotonic clock based on the system clock? If time.monotonic() is only sometimes available, then I don't see the added clock being anything more than an amusement. (In this case, I'd rather just use clock_gettime() and friends directly, because I have to be platform aware anyways.) What developers want is a timer that is useful for scheduling things to happen after predictable interval in the future, so we should give them that to the best of our ability. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On 3/28/2012 10:29 AM, Guido van Rossum wrote: On Wed, Mar 28, 2012 at 7:17 AM, Nick Coghlan ncogh...@gmail.com wrote: Completely unintuitive and unnecessary. With the GIL taking care of synchronisation issues, we can easily coerce time.time() into being a monotonic clock by the simple expedient of saving the last returned value: That's a pretty obvious trick. But why don't the kernels do this if monotonicity is so important? I'm sure there are also downsides, e.g. if the clock is accidentally set forward by an hour and then back again, you wouldn't have a useful clock for an hour. And the cache is not shared between processes so different processes wouldn't see the same clock value (I presume that most of these clocks have state in the kernel that isn't bound to any particular process -- AFAIK only clock() does that, and only on Unixy systems). What makes you think that isn't already true? I don't know what platforms that CPython compiles for that *won't* have one of the aforementioned functions available that provide a *real* monotonic clock. Surely, any platform that doesn't didn't recognize the need for it, or they would just provide a monotonic clock. That is to say, if you are a POSIX compliant system, then there is no reason to break gettimeofday() and friends when you can just implement CLOCK_MONOTONIC proper (even if it's just a trick like Nick's). I think the PEP should enumerate what platforms that CPython supports that will not benefit from a real monotonic clock. I think the number of platforms will be such a minority that the emulation makes sense. Practicality beats purity, and all. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Playing with a new theme for the docs, iteration 2
On 3/26/2012 1:00 AM, Greg Ewing wrote: This seems to be another case of the designer over-specifying things. The page should just specify a sans-serif font and let the browser choose the best one available. Or not specify a font at all and leave it up to the user whether he wants serif or sans-serif for the body text -- some people have already said here that they prefer serif. Why even bother formatting the page? The authorship and editorship have authority to dictate the presentation of the content. A large part of the effectiveness of a document and it's ease of consumption is determined by how it appears in whatever medium it's delivered on. While this particular medium invites the readership to participate in design choices, fonts are not all created equal and practical matters (size, tracking, and kerning) will dictate that some fonts will present better than other fonts. Consistent presentation across different systems is also a virtue, since people develop familiarity with the presentation and find information more readily if the presentation is consistent. I have no problem having Georg dictating to me the best font with which to present the documentation. However, I'd appreciate fallback choices that are of a similar appearance along the way to the ultimate fallback of sans-serif. Practically, the fonts available are unknown and unless we adopt the use of a liberally licensed OpenType font and use @font-face to embed a font, we need to provide fallbacks. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On 3/26/2012 7:32 PM, Victor Stinner wrote: I started to write the PEP 418 to clarify the notions of monotonic and steady clocks. time.steady This clock advances at a steady rate relative to real time. It may be adjusted. Please do not call this steady. If the clock can be adjusted, then it is not steady by any acceptable definition. I cannot fathom the utility of this function other than as a function that provides an automatic fallback from time.monotonic(). More importantly: this definition of steady is in conflict with the C++0x definition of steady that is where you sourced this named from![1] time.steady(strict=False) falls back to another clock if no monotonic clock is not available or does not work, but it does never fail. As I say above, that is so far away from what steady implies that this is a misnomer. What you are describing is a best-effort clock, which sounds a lot more like the C++0x high resolution clock. time.steady(strict=True) raises OSError if monotonic clock fails or NotImplementedError if the system does not provide a monotonic clock What is the utility of strict=True? If I wanted that mode of operation, then why would I not just try to use time.monotonic() directly? At worst, it generates an AttributeError (although that is not clear from your PEP). What is the use case for strict=True that is not covered by your time.monotonic()? If you want to define new clocks, then I wish you would use the same definitions that C++0x is using. That is: system_clock = wall clock time monotonic_clock = always goes forward but can be adjusted steady_clock = always goes forward and cannot be adjusted high_resolution_clock = steady_clock || system_clock Straying from that is only going to create confusion. Besides that, the one use case for time.steady() that you give (benchmarking) is better served by a clock that follows the C++0x definition. As well, certain kinds of scheduling/timeouts would be better implemented with the C++0x definition for steady rather than the monotonic one and vice-versa. Rather, it seems you have a particular use-case in mind and have settled on calling that a steady clock despite how it belies its name. [1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3128.html#time.clock.steady Objects of class steady_clock represent clocks for which values of time_point advance at a steady rate relative to real time. That is, the clock may not be adjusted. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 418: Add monotonic clock
On 3/26/2012 10:59 PM, Matt Joiner wrote:
So does anyone care to dig into the libstd++/boost/windoze
implementation to see how they each did steady_clock?
The Boost implementation can be summarized as:
system_clock:
mac = gettimeofday
posix = clock_gettime(CLOCK_REALTIME)
win = GetSystemTimeAsFileTime
steady_clock:
mac = mach_absolute_time
posix = clock_gettime(CLOCK_MONOTONIC)
win = QueryPerformanceCounter
high_resolution_clock:
* = { steady_clock, if available
system_clock, otherwise }
Whether or not these implementations meet the specification is an
exercise left to the reader..
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Playing with a new theme for the docs, iteration 2
On 3/25/2012 8:37 PM, Steven D'Aprano wrote: E.g. change the font-family from font-family: 'Lucida Grande',Arial,sans-serif; to font-family: 'Lucida Grande','Lucida Sans Unicode','Lucida Sans',Tahoma,Verdana,Arial,sans-serif; or similar. +1 To providing other fallbacks. As Steven says, on my Win7 machine, I do not have 'Lucida Grande' and it wasn't until he mentioned this that I compared the experience of the site with my MacBook (which looks much better!). This machine has both 'Lucida Sans Unicode' and 'Lucida Sans' and it's a toss-up to me which is better -- one is better than the other in certain contexts. Presumably, the character coverage of the Unicode font makes it the superior choice. Personally, I would leave Tahoma out of the list -- the kerning of the font is really aggressive and I find it much harder to read than Verdana. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/13/2012 9:57 PM, VanL wrote:
On Mar 13, 2012, at 8:37 PM, Martin v. Löwis mar...@v.loewis.de wrote:
The installation will end up in
c:\python33\lib\python3.3
which has the software name and version twice in the path.
Do we *really* need this?
We *already* have this. The only difference in this proposal is that we go
from py_version_nodot to py_version_short, i.e. from
c:\python33\lib\python33
to
c:\python33\lib\python3.3
Given that we already repeat it, isn't it better to be consistent?
Is it?
I think you are confusing two different configuration sections in
sysconfig.cfg:
[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}
[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 408 -- Standard library __preview__ package
On 1/29/2012 4:39 PM, Gregory P. Smith wrote: An example of this working: ipaddr is ready to go in. It got the eyeballs and API modifications while still a pypi library as a result of the discussion around the time it was originally suggested as being added. I or any other committers have simply not added it yet. This is wrong. PEP 3144 was not pronounced upon, so ipaddr is not just waiting for someone to commit it; it's waiting on consensus and pronouncement. PEP 3144 wasn't pronounced upon because there were significant disagreements about the design of the API proposed in the PEP. As it stands, I believe the authorship of ipaddr either decided that they were not going to compromise their module or lost interest. See Nick Coghlan's summary: http://mail.python.org/pipermail//python-ideas/2011-August/011305.html -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 408 -- Standard library __preview__ package
On 1/27/2012 8:48 PM, Barry Warsaw wrote: The thinking goes like this: if you would normally use an __preview__ module because you can't get approval to download some random package from PyPI, well then your distro probably could or should provide it, so get it from them. That is my thought about the entire __preview__ concept. Anything that would/should go into __preview__ would be better off being packaged for a couple of key distros (e.g., Ubuntu/Fedora/Gentoo) where they would get better visibility than just being on PyPI and would be more flexible in terms of release schedule to allow API changes. If the effort being put into making the __preview__ package was put into packaging those modules for distros, then you would get the same exposure with better flexibility and a better maintenance story. The whole idea of __preview__ seems to be a workaround for the difficult packaging story for Python modules on common distros -- stuffing them into __preview__ is a cheat to get the distro packagers to distribute these interesting modules since we would be bundling them. However, as you have pointed out, it would very desirable to them to not do so. So in the end, these modules may not receive as wide of visibility as the PEP suggests. I could very easily imagine the more stable distributions refusing or patching anything that used __preview__ in order to eliminate difficulties. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 7 clarification request: braces
On 1/1/2012 11:44 PM, Nick Coghlan wrote:
I think it's a recipe for future maintenance hassles when someone adds
a second statement to one of the clauses but doesn't add the braces.
(The only time I consider it reasonable to leave out the braces is for
one liner if statements, where there's no else clause at all)
Could you explain how these two cases differ with regard to maintenance?
In either case, there are superfluous edits required if the original
author had used braces *always*. Putting a brace on one-liners adds only
a single line to the code -- just like in the if/else case. So, your
argument seems conflicted. Surely, you would think this is a simpler
edit to make and diff to see in a patch file:
if(cond) {
stmt1;
+ stmt2;
}
vs.
-if(cond)
+if(cond) {
stmt1;
+ stmt2;
+}
Also, the superfluous edits will wrongly attribute the blame for the
conditional to the wrong author.
--
Scott Dial
sc...@scottdial.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] memcmp performance
On 10/20/2011 5:08 PM, Antoine Pitrou wrote: Have you reported gcc's outdated optimization issue to them? Or is it already solved in newer gcc versions? I checked this on gcc 4.6, and it still optimizes memcmp/strcmp into a repz cmpsb instruction on x86. This has been known to be a problem since at least 2002[1][2]. There are also some alternative implementations available on their mailing list. It seems the main objection to removing the optimization was that gcc isn't always compiling against an optimized libc, so they didn't want to drop the optimization. Beyond that, I think nobody was willing to put in the effort to change the optimization itself. [1] http://gcc.gnu.org/ml/gcc/2002-10/msg01616.html [2] http://gcc.gnu.org/ml/gcc/2003-04/msg00166.html -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] check for PyUnicode_READY look backwards
On 10/7/2011 7:13 PM, Terry Reedy wrote: On 10/7/2011 10:06 AM, Nick Coghlan wrote: FWIW, I don't mind whether it's 0 or == -1, so long as there's a comparison there to kick my brain out of Python boolean logic mode. Is there any speed difference (on common x86/64 processors and compilers)? I would expect that ' 0' should be optimized to just check the sign bit and 'if n 0' to 'load n; jump-non-negative'. There are several different ways to express those operators depending on the context. If n is worth moving into a register, then 0 will get to use a test and it's fewer instruction bytes than a cmp, but otherwise, it is no better. So, there is a very special case where 0 is better, but I think you'd be hard-pressed to measure it against the noise. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 393 Summer of Code Project
On 8/24/2011 4:11 AM, Victor Stinner wrote: Le 24/08/2011 06:59, Scott Dial a écrit : On 8/23/2011 6:38 PM, Victor Stinner wrote: Le mardi 23 août 2011 00:14:40, Antoine Pitrou a écrit : - You could try to run stringbench, which can be found at http://svn.python.org/projects/sandbox/trunk/stringbench (*) and there's iobench (the text mode benchmarks) in the Tools/iobench directory. Some raw numbers. stringbench: 147.07 203.07 72.4 TOTAL for the PEP 393 146.81 140.39 104.6 TOTAL for default = PEP is 45% slower I ran the same benchmark and couldn't make a distinction in performance between them: Hum, are you sure that you used the PEP 383? Make sure that you are using the pep-383 branch! I also started my benchmark on the wrong branch :-) You are right. I used the Get Source link on bitbucket to save pulling the whole clone, but the Get Source link seems to be whatever branch has the lastest revision (maybe?) even if you switch branches on the webpage. To correct my previous post: cpython.txt 183.26 177.97 103.0 TOTAL cpython-wide-unicode.txt 181.27 195.58 92.7TOTAL pep-393.txt 181.40 270.34 67.1TOTAL And, cpython.txt real0m32.493s cpython-wide-unicode.txt real0m33.489s pep-393.txt real0m36.206s -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 393 Summer of Code Project
On 8/23/2011 6:38 PM, Victor Stinner wrote: Le mardi 23 août 2011 00:14:40, Antoine Pitrou a écrit : - You could try to run stringbench, which can be found at http://svn.python.org/projects/sandbox/trunk/stringbench (*) and there's iobench (the text mode benchmarks) in the Tools/iobench directory. Some raw numbers. stringbench: 147.07 203.07 72.4 TOTAL for the PEP 393 146.81 140.39 104.6 TOTAL for default = PEP is 45% slower I ran the same benchmark and couldn't make a distinction in performance between them: pep-393.txt 182.17 175.47 103.8 TOTAL cpython.txt 183.26 177.97 103.0 TOTAL pep-393-wide-unicode.txt 181.61 198.69 91.4TOTAL cpython-wide-unicode.txt 181.27 195.58 92.7TOTAL I ran it a couple times and have seen either default or pep-393 being up to +/- 10 sec slower on the unicode tests. The results of the 8-bit string tests seem to have less variance on my test machine. run test_unicode 50 times: 0m19.487s for PEP 0m17.187s for default = PEP is 13% slower $ time ./python -m test `python -c 'print test_unicode * 50'` pep-393-wide-unicode.txt real0m33.409s cpython-wide-unicode.txt real0m33.489s Nothing in it for me.. except your system is obviously faster, in general. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): #5301: add image/vnd.microsoft.icon (.ico) MIME type
On 8/21/2011 3:12 PM, Terry Reedy wrote: On 8/21/2011 5:09 AM, Sandro Tosi wrote: I can see your point: the reason I committed it also on the stable branches is that .ico are already out there (since a long time) and they were currently not recognized. I can call it a bug. But it is not (a behavior bug). Every feature request 'fixes' what its proposer considers to be a design bug or something. What's the feature added? That's a semantic game. Anyhow, if it was not appropriate, just tell me and I'll revert on 2.7 and 3.2 . Thanks for your input! It is a new feature for the same reason http://bugs.python.org/issue10730 was. If that had not been added for 3.2.0 (during the beta period, with Georg's permission), it would have waited for 3.3.s ISTM, that Issue #10730 was more contentious because it is *not* an IANA-assigned mime-type, whereas image/vnd.microsoft.icon is and has been since 2003. Whereas image/svg+xml didn't get approved until earlier this month, AFAICT. Our intent is that the initial CPython x.y.0 release 'freeze' the definition of Python x.y. Code that were to use the new feature in 3.2.3 would not work in 3.2.0,.1,.2, making 3.2.3 define a slight variant. People who want the latest version of an stdlib module should upgrade to the latest release or even download from the repository. For mimetypes, the database can be explicitly augmented in the code and then the code would work in all 2.7 or 3.2 releases. Doesn't that weaken your own argument that changing the list in Lib/mimetypes.py doesn't violate the freeze? Considering that the mime-types are automatically read from a variety of out-of-tree locations? It's already the case that the list of mime-types recognized by a given CPython x.y.z is inconsistent from platform-to-platform and more importantly installation-to-installation (since /etc/mime.types could be customized by a given distribution or modified by a local administrator, and on Windows, the mime-types are scrapped from the registry). On any reasonable system that I can get access to at the moment (Gentoo, OS X, Win7), '.ico' is already associated with 'image/x-icon' via either scrapping the /etc/mime.types or the registry. I think this issue probably originates with CPython 2.6 on Windows, where there was no help from the registry or external mime.types file. Nevertheless, I am +0 for adding entries from the IANA list into stable versions because I don't see how they could ever harm anyone. Any robust program would need to be responsible and populate the mimetypes itself, if it depended on them, otherwise, all bets are off about what types_map contains from run-to-run of a program (because /etc/mime.types might have changed). -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): Skip test_getsetlocale_issue1813() on Fedora due to setlocale() bug.
On 8/2/2011 4:48 AM, Stefan Krah wrote: R. David Murray rdmur...@bitdance.com wrote: If there are any helpful tests I can run on Gentoo, please let me know. Yes, you could run the small test program. If you get the same results as on Fedora, then I wonder why the Gentoo buildbots are green. Do they have tr_TR and tr_TR.iso88599 installed? Highly doubtful. It is a normal part of the Gentoo install process to select the locales that you want for the system. Even the example list of locales doesn't include any Turkish locales, so one would've had to gone to specific effort to add that one. -- Scott Dial sc...@scottdial.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Supporting Visual Studio 2010
On 4/4/2011 6:43 PM, Michael Foord wrote: Won't that still be an issue despite the stable ABI? Extensions on Windows should be linked to the same version of MSVCRT used to compile Python - and every time we switch version of Visual Studio it is usually accompanied by a switch in MSVCRT version. My understanding (but I haven't looked closely) was that the stable ABI specifically excluded anything that would expose a problem due to a CRT mismatch -- making this a moot point. I'm sure Martin will correct me if I am wrong. -Scott -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Impaired Usability of the Mercurial Source Viewer
On 3/31/2011 8:30 PM, Antoine Pitrou wrote: On Thu, 31 Mar 2011 16:15:48 -0700 Raymond Hettinger raymond.hettin...@gmail.com wrote: The Hg source viewer needs to be tweaked to improve its usability. What we've got now is a step backwards from the previous svn viewer. Looking at http://hg.python.org/cpython/file/default/Lib/linecache.py for example, there are two issues. 1) the code cannot be cut-and-pasted because the line numbers are commingled with the source text. 2) the code is hard to read because of the alternating white and gray bars. Contrast that to the more typical, beautiful presentations with a solid background and the ability to cut-and-paste without grabbing line numbers: This is something you need to discuss with the Mercurial project. See http://mercurial.selenic.com/bts/ and http://mercurial.selenic.com/wiki/ContributingChanges The hgweb interface is templated. You can already change it via style in the hgweb.conf. There are several styles already available in the templates folder of the install, and you could provide your own if you like too. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP and reference implementation of a Python launcher for Windows
On 3/19/2011 7:23 PM, Nick Coghlan wrote: On Sun, Mar 20, 2011 at 9:04 AM, Martin v. Löwis mar...@v.loewis.de wrote: Am 19.03.2011 23:51, schrieb Nick Coghlan: On Sun, Mar 20, 2011 at 4:49 AM, Martin v. Löwis mar...@v.loewis.de wrote: I, for example, will find issues with it if the implementation uses CreateProcess at some point - the launcher should IMO run Python directly, rather than creating a new process. This, in turn, will probably mean that you cannot actually implement the PEP in Python. runpy performs basically that dance to implement _run_module_as_main(), so it's definitely possible. Does it also support loading a different interpreter version than the one it is running? Ouch, no. I guess you're right, any Python based solution will need to create a child process. Why would that be true? Shouldn't this launcher just be a basic wrapper that cobbles together the arguments for an eventual os.exec*() call? What is there to do other than to exec the correct interpreter with (a subset of?) the command-line arguments? -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bugs in thread_nt.h
On 3/10/2011 3:07 AM, Paul Du Bois wrote: volatile considered harmful http://www.kernel.org/doc/Documentation/volatile-considered-harmful.txt -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] hg mq workflow is broken (issue11450)
I wanted to draw attention to issue11450 [1]. In trying to using mq to work on patches for CPython, I found that I could no longer get regrtest to run. This issue comes down to the size of the output of hg id -t . being much longer than expected. $ hg qnew dummy-patch $ echo /* dummy change */ Modules/getbuildinfo.c $ ./python -c 'import sys; print(sys.version)' 3.3a0 (dummy-patch qbase qtip tip:191f047a07b4+, Mar 9 ) [GCC 4.4.5] This is a showstopper for using mq and CPython, since you can no longer run regrtest because the platform module cannot parse that truncated string. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] hg mq workflow is broken (issue11450)
On 3/9/2011 3:15 AM, Scott Dial wrote: I wanted to draw attention to issue11450 [1]. In trying to using mq to work on patches for CPython, I found that I could no longer get regrtest to run. Just to update this thread, thanks to the swift work of Nadeem Vawda and Antoine for pushing it in, it's fixed! -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate the faulthandler module into Python 3.3?
On 3/4/2011 6:10 AM, Nick Coghlan wrote:
On Fri, Mar 4, 2011 at 10:58 AM, Victor Stinner
victor.stin...@haypocalc.com wrote:
So, what do you think?
Something we may want to consider is enabling it by default in
interactive mode, and also when `-i` is specified on the command line.
I am still bothered by the fact that,
import faulthandler
faulthandler.enable()
import sys
sys.stderr.close()
sys.stderr = open('logs/error.log', 'wb')
faulthandler.sigsegv()
, does the wrong thing. In this incantation, it's easy to say that it's
programmer error, but I think this still precludes it from being on by
default (where the first two statement are implicitly executed by the
interpreter). It's probably uncommon enough to close stderr from an
interactive interpreter session that it doesn't bother me (although I am
not sure the utility of that), but I would hesitate to say that is true
for using '-i'.
Otherwise, the functionality seems generally useful and it's on my list
of things to integrate into my application, and having it in the stdlib
is one less external dependency for me to manage.
-Scott
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] devguide (hg_transition): Advertise hg import over patch.
On 2/27/2011 10:18 AM, Antoine Pitrou wrote: Well, chances are TortoiseHG comes with an UI to apply patches (TortoiseSVN had one), so the command-line instructions may be of little use to them. I don't believe TortoiseHG has such a feature (or I can't find it), although if you have TortoiseSVN, you can still use that as a patch tool. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On 2/24/2011 4:02 AM, M.-A. Lemburg wrote: I get 788,000 hits for 'latin1 -latin-1' on Google, 'latin-1' gives 2,600,000 hits. Looks like it's still the preferred way to write that encoding name. That's bogus. You can't search for latin-1 on Google, it isn't strict enough. The third hit is a url containing latin1 and a title of Latin 1. And it picks up things like Latin 1: The Easy Way, which is a book on Latin. However, you *can* search much more strictly on Google Code Search, which gives 4,014 (latin-1) to 13,597 (latin1). http://www.google.com/codesearch?hl=enlr=q=%28\%22latin1\%22|\%27latin1\%27%29sbtn=Search http://www.google.com/codesearch?hl=enlr=q=%28\%22latin-1\%22|\%27latin-1\%27%29sbtn=Search So, no, I don't think the development world aligns with your pedantry. That's not to say this is a popularity contest, but then let's not cite google hit counts as proof. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On 2/23/2011 9:19 PM, Alexander Belopolsky wrote: On Wed, Feb 23, 2011 at 8:48 PM, Dj Gilcrease digitalx...@gmail.com wrote: Google Code search limited to python latin1: 3,489 http://www.google.com/codesearch?hl=enlr=q=latin1+lang%3Apythonsbtn=Search latin-1: 5,604 http://www.google.com/codesearch?hl=enlr=q=latin-1+lang%3Apythonsbtn=Search latin1: 1,618 http://www.google.com/codesearch?hl=enlr=q=%28\%22latin1\%22|\%27latin1\%27%29+lang%3Apythonsbtn=Search latin-1: 2,241 http://www.google.com/codesearch?hl=enlr=q=%28\%22latin-1\%22|\%27latin-1\%27%29+lang%3Apythonsbtn=Search utf8: 25,341 http://www.google.com/codesearch?hl=enlr=q=utf8+lang%3Apythonsbtn=Search utf-8: 179,806 http://www.google.com/codesearch?hl=enlr=q=utf-8+lang%3Apythonsbtn=Search utf8 9,676 http://www.google.com/codesearch?hl=enlr=q=%28\%22utf8\%22|\%27utf8\%27%29+lang%3Apythonsbtn=Search utf-8 44,795 http://www.google.com/codesearch?hl=enlr=q=%28\%22utf-8\%22|\%27utf-8\%27%29+lang%3Apythonsbtn=Search Your search is invalid. You hit things such as Latin1ClassModel which have no relevance to the issue at hand. You get about the same ratio if you filter out only the quoted strings. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #11051: system calls per import
On 1/31/2011 1:38 PM, Brett Cannon wrote: I should mention that I have considered implementing a caching finder and loader for filesystems in importlib for people to optionally install to use for themselves. The real trick, though, is should it only cache hits, misses, or both? Regardless, though, it would be a very simple mixin or subclass to implement if there is demand for this sort of thing. I have in the past implemented a PEP302 finder/loader zipfile-based cache. On campus, I use a version of python installed to my home directory that is on an NFS share. I found such a cache often gave slower startup times for applications like bzr and hg. My cache merely stores things it finds things in sys.path and loads from the zipfile names that it knows and storing those that it doesn't. I make no attempt to invalidate the cache contents once stored. So, I am already talking about a best-case scenario for caching. I'm not sure how you could invalidate the cache without paying the cost of all the normal syscalls that we are trying to avoid. My finder/loader is not bug-free, but I'd be glad to make it available to someone if they want to play around with it. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exception __name__ missing?
On 1/17/2011 3:22 PM, Ron Adam wrote: Is this on purpose? This reminds me of something I ran into a few years ago wrt. the attribute on exceptions. Namely, that instances of built-in exceptions do not have a __module__ attribute, but instance of user exceptions do -- a change which appeared in Python 2.5: http://mail.python.org/pipermail/python-list/2007-November/1088229.html I had a use case, using ZSI to provide a SOAP interface, where being able to get the __module__ and __name__ was needed (to serialize into a SOAP fault message). I worked around the issue by referencing the __class__ (as the other replier mentioned). But, I didn't receive any responses then, so I think not a lot of attention was put into these type of attributes on exceptions. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Search-friendly shortcuts for Windows?
On 12/20/2010 8:38 PM, Nick Coghlan wrote: would it be reasonable to consider including the major version number in the start menu shortcut names? +1 First thing I did was add an x.y prefix to the Python shortcuts. There are a lot of application shortcuts on my Win7 system that have version numbers that have no naming conflicts, so even if a single version of Python was installed, it would not look out of place at all. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable
On 12/18/2010 8:50 AM, James Y Knight wrote: I think instead of calling abort() to kill the process, you should: - install the signal handler with SA_NODEFER|SA_RESETHAND (or if sigaction is not present, explicitly reset the action to SIG_DFL and unblock first thing upon entering the handler), and then, - at the end of the handler, kill(getpid(), orig_signal) in order to abort the process. I concur with this being the correct way to right such a handler. This has two advantages: 1) the process's exit code will actually show the correct signal, It's more than an advantage: it's the only correct way to handle a termination signal. 2) it might let the OS fault handlers work properly as well -- I'm not sure. If it does, you may want to experiment with whether having or omitting SA_NODEFER gives a better backtrace (from the OS mechanism) in that case. Even if that doesn't work, things like the grsecurity patches to linux use these signals to detect exploits and log them and do throttling. Calling abort() effectively converts all of these faults into SIGABRT terminations that are considered (more?) innocent terminations. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] futures API
On 12/11/2010 9:44 AM, Thomas Nagy wrote: The amount of work items processed by unit of time does not seem to be a straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out of curiosity, what is the _thread_references for? The source file for the example is in: http://www.freehackers.org/~tnagy/futures_test3.py The diagram was created by: http://www.freehackers.org/~tnagy/futures_test3.plot You're test code does 50,000,000 of list appends. I suspect your benchmark is telling you more about the behavior of large lists than the overhead of the futures module. You should retry that experiment with the list pre-allocated. Beyond that, the curve in that line is not exactly a large amount of variance from a straight line. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 11/7/2010 7:09 PM, Martin v. Löwis wrote: Luckily, the problems that we faced 2.5 years ago when I came up with the idea of Snakebite are still just as ever present today ;-) Is this bashing of existing infrastructure really necessary? People (like me) might start bashing about vaporware and how a bird in the hand is worth two in the bush. Cooperate, don't confront. +1 Respect your (software) elders. The Snaketbite rhetoric has always been less than generous with regard to Buildbot, but Buildbot has been providing an infinitely more useful service to the community for much longer than Snakebite has for those 2.5 years. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)
On 11/7/2010 9:58 PM, C. Titus Brown wrote: Yes, yes, I agree that some graciousness is a good idea. Oh, wait... you're not helping. Classy. I don't remember being invited to help. snakebite.org is a dead end. snakebite-list hasn't had a post for over a year. Where is the list of things that you need done so that I can get started on that? Oh wait.. Seriously, all I asked was for you to tone down your insults to a technology that is already solving problems today. Why you feel the need to attack me personally is beyond my understanding. Furthermore, I don't see why I need to be helping -- somebody who doesn't want help -- to be able to deduce that your message is being insulting to the authors of Buildbot. On 11/7/2010 9:58 PM, C. Titus Brown wrote: And I'm not in a personal position to help, so I've basically tried to shut up about it :). At least I am in good company. ;) -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r85934 - in python/branches/py3k: Misc/NEWS Modules/socketmodule.c
On 10/30/2010 4:08 PM, Martin v. Löwis wrote: I think size should be in TCHARs, not in bytes. (MSDN says so) And GetComputerName's signature differs from MSDN. (Maybe should use GetComputerNameExW again?) You are right. So how about this patch? Still not quite right. The call to GetComputerNameExW after ERROR_MORE_DATA (which gives the number of *bytes* needed) still needs to pass size/sizeof(wchar_t) back into GetComputerNameExW since it wants the number TCHARs. I don't think the +1 is needed either (MSDN says it already included the null-terminator in the byte count. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r85838 - python/branches/py3k/.gitignore
On 10/27/2010 3:25 AM, anatoly techtonik wrote: On Tue, Oct 26, 2010 at 3:51 PM, Barry Warsaw ba...@python.org wrote: On Oct 26, 2010, at 09:19 PM, Nick Coghlan wrote: On Tue, Oct 26, 2010 at 5:31 PM, Georg Brandl g.bra...@gmx.net wrote: This looks more like Add gitignore. Do we really want to check in ignore files for every possible DVCS? No, but supporting the current big four open source ones (svn, hg, bzr, git) seems reasonable enough. +1. A couple of extra dot files never hurt anyone. :) Why hg and bzr can't be tuned to understand .svnignore files? Is there such a thing? I think you mean the svn:ignore metadata, which I doubt you can get to through either mirror. Even if you could, svn:ignore is pretty weak (only applies to the directory it is set on and can only do glob patterns). .hgignore is significantly more robust (and only has to be written to a single spot) since matches entire paths and supports regex. Eventually, we'll need to build an appropriate .hgignore file for the python tree anyways. As with others, I don't see the harm in committers who use those tools adding and maintaining these files. Seems akin to having Misc/python-mode.el and Misc/Vim/python.vim. It's all in the spirit of supporting the tools that people are actually using. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Patch making the current email package (mostly) support bytes
On 10/2/2010 7:00 PM, R. David Murray wrote:
The clever hack (thanks ultimately to Martin) is to accept 8bit data
by encoding it using the ASCII codec and the surrogateescape error
handler.
I've seen this idea pop up in a number of threads. I worry that you are
all inventing a new kind of dual that is a direct parallel to Python 2.x
strings. That is to say,
3.x b = b'\xc2\xa1'
3.x s = b.decode('utf8')
3.x v = b.decode('ascii', 'surrogateescape')
, where s and v should be the same thing in 3.x but they are not due
to an encoding trick. I believe this trick generates more-or-less the
same issues as strings did in 2.x:
2.x b = '\xc2\xa1'
2.x s = b.decode('utf8')
2.x v = b
Any reasonable 2.x code has to guard on str/unicode and it would seem in
3.x, if this idiom spreads, reasonable code will have to guard on
surrogate escapes (which actually seems like a more expensive test). As in,
3.x print(v)
Traceback (most recent call last):
File stdin, line 1, in module
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc2' in
position 0: surrogates not allowed
It seems like this hack is about making the 3.x unicode type more like
the 2.x string type, and I thought we decided that was a bad idea. How
will developers not have to ask themselves whether a given string is a
real string or a byte sequence masquerading as a string? Am I missing
something here?
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/25/2010 5:37 PM, Martin v. Löwis wrote: Unfortunately, most sites using OpenID seem have an awkward login process. Maybe it's just me (I don't use OpenID much) but I expect that without a lot more handholding of new users, OpenID actually turns more people away than any other registration/login process. So how do you like the OpenID login of PyPI, which has a Google, MyOpenID and Launchpad icon, which users need to click on to create in account or login? The ultra geeks demanded and got a separate page where they can enter long URLs. Having just tried this out. A few comments: 1) Registering via OpenID is a bit clumsy since there is a Register link that does not mention OpenID. 2) The URL registered with the OpenID provider is a bit of a wart: http://pypi.python.org/pypi?:action=openid_return; vs. http://bitbucket.org/; 3) The email I received asked me to Complete your Cheese Shop registration which I think is just an oversight since the relabeling to pypi. 4) It's a bit clumsy that Login pops up an HTTP Authentication prompt, which is useless to someone who only has never set a password and relies only on an OpenID credential. Furthermore, the 401 page does not provide a quick way to get to use OpenID. In general, I am pretty happy with pypi's support of OpenID considering it allowed me to use my own provider, which often has not been the case with other sites. My experience with trying to adopt OpenID as a way of life has been poor mostly because many sites fail to support anything but a few major OpenID providers (e.g., Google). I appreciate has a fast-path for those providers and yet let's me still use my own. Although, I think it would be nice if I didn't have to go to another page to do that, but I may be biased by having such a short OpenID URI. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/26/2010 3:12 AM, Martin v. Löwis wrote: 2) The URL registered with the OpenID provider is a bit of a wart: http://pypi.python.org/pypi?:action=openid_return; vs. http://bitbucket.org/; You mean, as this is what the provider then shows you for confirmation? The provider also lists the trusted sites by these return URLs, and that is where I saw it as being a bit of a wart. I use the OpenID plugin for WordPress as my provider, so it may be that it doesn't do this correctly. I noticed that Google shows just pypi.python.org, but the WordPress plugin shows that return URL instead. Nevertheless, I agree that it's too late/not worth it to change that now. I think there is no way out wrt. to the basic auth prompt. I could label the Login link Password login if you think this would help. The basic auth prompt doesn't bother me so much as the fact that the 401 doesn't have a Use OpenID [Google] [myOpenID] [Launchpad] set of links; you have to use the brower's back button because the only links offered are to register or reset your password. Preventing the browser from prompting the user on the chance they might want to enter an OpenID is not possible, and stopping to use basic authentication is not feasible. In theory, you could catch usernames that started with http://;, but I imagine that only ultra geeks know their URIs (I have no idea what the URI for a Google account is). But, I don't see this as being worthwhile either; I just think it would be nice if the 401 page gave a quick way to correct one's mistake that didn't involve the back button. And again, enjoying a short OpenID URI probably does put you in the ultra geek category (which I seriously don't mean as an offense). No offense taken. :) -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python wiki
On 9/26/2010 11:45 PM, R. David Murray wrote: On Sun, 26 Sep 2010 21:56:20 -0400, Scott Dial scott+python-...@scottdial.com wrote: On 9/26/2010 3:12 AM, Martin v. Loewis wrote: Preventing the browser from prompting the user on the chance they might want to enter an OpenID is not possible, and stopping to use basic authentication is not feasible. In theory, you could catch usernames that started with http://;, but I No, Martin really meant not possible: once basic auth is started, the browser prompts for username and password and you are in basic-auth land thereafter; the web server has *no way* to tell the browser to *stop* using basic auth. I agree that once you reply with a 401 that you will prompt the user, but my point was what username means in the Authorization header is open to interpretation by the HTTP server and/or script handling the GET request. imagine that only ultra geeks know their URIs (I have no idea what the URI for a Google account is). But, I don't see this as being worthwhile Well, my OpenId is 'david.bitdance.com', so even if you could get around the basic auth problem, looking for http://; wouldn't work. That's actually not a valid OpenID[1], but the OpenID specification says a relaying party MUST normalize identifiers[2] (in this case, prepending the http://;). I believe bugs.python.org does this by checking for a username first(?), and failing any matches, it normalizes it for OpenID discovery. Otherwise, I can always use the canonical form of my ID http://scottdial.com; to login (assuming ':' and '/' are illegal characters for usernames). I say all this not with the intent of saying pypi *needs* this, but to refute the notion that OpenID must be clumsy to use. [1] http://openid.net/specs/openid-authentication-2_0.html Identifier: An Identifier is either a http or https URI, (commonly referred to as a URL within this document), or an XRI (Reed, D. and D. McAlpin, “Extensible Resource Identifier (XRI) Syntax V2.0,” .) [XRI_Syntax_2.0]. [2] http://openid.net/specs/openid-authentication-2_0.html#normalization 3. Otherwise, the input SHOULD be treated as an http URL; if it does not include a http or https scheme, the Identifier MUST be prefixed with the string http://;. If the URL contains a fragment part, it MUST be stripped off together with the fragment delimiter character #. See Section 11.5.2 (HTTP and HTTPS URL Identifiers) for more information. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Return from generators in Python 3.2
On 8/26/2010 11:00 AM, Yury Selivanov wrote: If we decide to postpone this feature till Python 3.3, than we'll push it all back The change is tiny, but it means really a lot. AFAICT, this change was the most controversial part of PEP 380. PS I'm attaching a patch to the letter; it's far from ideal state, but contains the GeneratorReturn exception, code to raise it and the corresponding unittests. I believe overloading StopIteration for this purpose was considered more acceptable than creating a new exception. BTW, attaching patches to emails on this list is generally the best way to have few look at your patch. :-p Also, this seems more appropriate for python-ideas. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/30/2010 2:53 PM, Barry Warsaw wrote: It might be amazing, but it's still a significant overhead. As I've described, multiply that by all the py files in all the distro packages containing Python source code, and then still try to fit it on a CDROM. I decided to prove to myself that it was not a significant issue to have parallel directory structures in a .tar.bz2, and I was surprised to find it much worse at that then I had imagined. For example, # cd /usr/lib/python2.6/site-packages # tar --exclude=*.pyc --exclude=*.pyo \ -cjf mercurial.tar.bz2 mercurial # du -h mercurial.tar.bz2 640Kmercurial.tar.bz2 # cp -a mercurial mercurial2 # tar --exclude=*.pyc --exclude=*.pyo \ -cjf mercurial2.tar.bz2 mercurial mercurial2 # du -h mercurial.tar.bz2 1.3Mmercurial2.tar.bz2 So, I was definitely wrong in saying that you do better than doubling. What happens to the distro packaging if a python package splits the codebase between 2.x and 3.x (meaning they have distinct .py files)? The Debian/Ubuntu approach to Python 2/3 support is to provide them in separate distro packages. E.g. for Python package foo, you would have Debuntu package python-foo (for the Python 2.x version) and python3-foo. We do not share source between Python 2 and 3 versions, at least not yet wink. I had asked this question to point out that you will still need to accommodate some form of version-specific packages (I am not a debuntu expert by any means). And, I think your response is an acknowledgment of that fact, however it's certainly true that there are few examples, as you said. I appreciate all your replies. I am not sure a PEP is really needed here, but to having had all of this discussed and explained on the mailing list is certainly useful. I trust that yourself and the debuntu python group will end up chasing down and taking care of any quirks that this change might cause, so I am not worried about it. :D -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/26/2010 4:06 PM, Matthias Klose wrote: On 25.06.2010 22:12, James Y Knight wrote: On Jun 25, 2010, at 4:53 AM, Scott Dial wrote: Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. This is a good point, but I think still falls short of a solution. For a package like lxml, indeed you are correct. Since debian needs to build it once per version, it could just put the entire package (.py files and .so files) into a different per-python-version directory. This is what is currently done. This will increase the size of packages by duplicating the .py files, or you have to install the .py in a common location (irrelevant to sys.path), and provide (sym)links to the expected location. This is what is currently done and provide (sym)links to the expected location are conflicting statements. If you are symlinking .py files from a shared location, then that is not the same as just install the package into a version-specific location. What motivation is there for preferring symlinks? Who cares if a ditro package install yields duplicate .py files? Nor am I motivated by having to carry duplicate .py files in a distribution package (I imagine the compression of duplicate .py files is amazing). A different per-python-version directory also has the disadvantage that file conflicts between (distribution) packages cannot be detected. Why? That sounds like a broken tool, maybe I am naive, please explain. If two packages install /usr/lib/python2.6/foo.so that should be just as detectable two installing /usr/lib/python-shared/foo.cpython-26.so If you *must* compile .so files for every supported version of python at packaging time, then you are already saying the set of python versions is known. I fail to see the difference between a package that installs .py and .so files into many directories than having many .so files in a single directory; except that many directories *already* works. The only gain I can see is that you save duplicate .py files in the package and on the filesystem, and I don't feel that gain alone warrants this fundamental change. I would appreciate a proper explanation of why/how a single directory is better for your distribution. Also, I haven't heard anyone that wasn't using debian tools chime in with support for any of this, so I would like to know how this can help RPMs and ebuilds and the like. I don't think that installation into different locations based on the presence of extension will work. Should a location really change if an extension is added as an optimization? Splitting a (python) package into different installation locations should be avoided. I'm not sure why changing paths would matter; any package that writes data in its install location would be considered broken by your distro already, so what harm is there in having the packaging tool move it later? Your tool will remove the old path and place it in a new path. All of these shenanigans seem to manifest from your distro's python-support/-central design, which seems to be entirely motivated by reducing duplicate files and *not* simplifying the packaging. While this plan works rather well with .py files, the devil is in the details. I don't think Python should be getting involved in what I believe is a flawed design. What happens to the distro packaging if a python package splits the codebase between 2.x and 3.x (meaning they have distinct .py files)? As someone else mentioned, how is virtualenv going to interact with packages that install like this? -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 8:23 PM, James Y Knight wrote: On Jun 24, 2010, at 5:53 PM, Scott Dial wrote: If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared) Because python looks for .so files in the same place it looks for the .py files of the same package. My suggestion was that a package that contains .so files should not be shared (e.g., the entire lxml package should be placed in a version-specific path). The motivation for this PEP was to simplify the installation python packages for distros; it was not to reduce the number of .py files on the disk. Placing .so files together does not simplify that install process in any way. You will still have to handle such packages in a special way. You must still compile the package multiple times for each relevant version of python (with special tagging that I imagine distutils can take care of) and, worse yet, you have created a more trick install than merely having multiple search paths (e.g., installing/uninstalling lxml for *one* version of python is actually more difficult in this scheme). Either the motivation for this PEP is inaccurate or I am failing to understand how this is *simpler*. In the case of pure-python, this PEP is clearly a win, but I have not seen an argument that it is a win for .so files. Moreover, the PEP itself is titled PYC Repository Directories (not shared site-packages) and makes no mention of .so files at all. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 9:18 PM, Greg Ewing wrote: Scott Dial wrote: But the only motivation for doing this with .pyc files is that the .py files are able to be shared, In an application made up of a mixture of pure Python and extension modules, the .py files are able to be shared too. Seems to me that a similar motivation exists here as well. Not exactly the same, but closely related. If I recall Barry's motivation correctly, the PEP was intended to simplify the installation of packages for multiple versions of Python, although the PEP states that in a less direct way. In the case of pure-python packages, this is merely about avoiding .pyc collisions. But, in the case of packages with .so files, I fail to see how this is simpler (in face, I believe it to be more complicated). So, I am not sure the PEP supports this feature being proposed (since it makes no mention of .so files), and more importantly, I am not sure it actually makes anything better for anyone (still requires multiple compilations and un/install gymnastics). -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/25/2010 2:58 PM, Brett Cannon wrote: I assume you are talking about PEP 3147. You're right that the PEP was for pyc files and that's it. No one is talking about rewriting the PEP. Yes, I am making reference to PEP 3147. I make reference to that PEP because this change is of the same order of magnitude as the .pyc change, and we asked for a PEP for that, and if this .so stuff is an extension of that thought process, then it should either be reflected by that PEP or a new PEP. The motivation Barry is using is an overarching one of distros wanting to use a single directory install location for all installed Python versions. That led to PEP 3147 and now this work. It's unclear to me that that is the correct motivation, which you are divining. As I understand it, the motivation to be to *simplify installation* for distros, which may or may not be achieved by using a single directory. In the case of pure-python packages, a single directory is an obvious win. In the case of mixed-python packages, I remain to be persuaded there is any improvement achieved. This is meant to be used by distros in a programmatic fashion, so my response is so what? Their package management system is going to maintain the directory, not a person. Then why is the status quo unacceptable? I have already explained how this will still require programmatic steps of at least the same difficulty as the status quo requires, so why should we change anything? I am skeptical that this is a simple programmatic problem either: take any random package on PyPI and tell me whether or not it has a .so file that must be compiled. If such a .so file exists, then this package must be special-cased and compiled for each version of Python on the system (or will ever be on the system?). Such a package yields an arbitrary number of .so files due to the number of version of Python on the machine, and I can't imagine how it is simpler to manage all of those files than it is to manage multiple site-packages. You're conflating what is being discussed with PEP 3147. That PEP is independent of this. PEP 3147 just empowered this work to be relevant. Without a PEP (be it PEP 3147 or some other), what is the justification for doing this? The burden should be on you to explain why this is a good idea and not just a clever idea. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] versioned .so files for Python 3.2
On 6/24/2010 5:09 PM, Barry Warsaw wrote: What use case does this address? Specifically, it's the use case where we (Debian/Ubuntu) plan on installing all Python 3.x packages into /usr/lib/python3/dist-packages. As of PEP 3147, we can do that without collisions on the pyc files, but would still have to symlink for extension module .so files, because they are always named foo.so and Python 3.2's foo.so won't (modulo PEP 384) be compatible with Python 3.3's foo.so. If the package has .so files that aren't compatible with other version of python, then what is the motivation for placing that in a shared location (since it can't actually be shared)? So using the same trick as in PEP 3147, if we can name Python 3.2's foo extension differently than the incompatible Python 3.3's foo extension, we can have them live in the same directory without symlink tricks. Why would a symlink trick even be necessary if there is a version-unspecific directory and a version-specific directory on the search path? PEP 3147 addresses the fact that the user may have different versions of Python installed and each wants to write a .pyc file when loading a module. .so files are not generated simply by running the Python interpreter, ergo .so files are not an issue for that use case. See above. It doesn't matter whether the pyc or so is created at run time by the user or by the distro build system. If the files for different Python versions end up in the same directory, they must be named differently too. But the only motivation for doing this with .pyc files is that the .py files are able to be shared, since the .pyc is an on-demand-generated, version-specific artifact (and not the source). The .so file is created offline by another toolchain, is version-specific, and presumably you are not suggesting that Python generate it on-demand. If you want to make it so a system can install a package in just one location to be used by multiple Python installations, then the version number isn't enough. You also need to distinguish debug builds, profiling builds, Unicode width (see issue8654), and probably several other ./configure options. This is a good point, but more easily addressed. Let's say a distro makes three Python 3.2 variants available, one normal build, a debug build, and UCS2 and USC4 versions of the above. All we need to do is choose a different .so ABI tag (see previous follow) for each of those builds. My updated patch (coming soon) allows you to define that tag to configure. So e.g. Why is this use case not already addressed by having independent directories? And why is there an incentive to co-mingle these version-punned files with version-agnostic ones? Mix and match for any other build options you care about. Because the distro controls how Python is configured, this should be fairly easy to achieve. For packages that have .so files, won't the distro already have to build multiple copies of that package for all version of Python? So, why can't it place them in separate directories that are version-specific at that time? This is not the same as placing .py files that are version-agnostic into a version-agnostic location. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] variable name resolution in exec is incorrect
On 5/27/2010 7:14 AM, Colin H wrote:
def define_stuff(user_code):
context = {...}
stuff = {}
stuff.update(context)
exec(user_code, stuff)
return_stuff = {}
return_stuff.update(stuff)
del return_stuff['__builtins__']
for key in context:
if key in return_stuff and return_stuff[key] == context[key]:
del return_stuff[key]
return return_stuff
I'm not sure your application, but I suspect you would benefit from
using an identity check instead of an __eq__ check. The equality check
may be expensive (e.g., a large dictionary), and I don't think it
actually is checking what you want -- if the user_code generates an
__eq__-similar dictionary, wouldn't you still want that? The only reason
I can see to use __eq__ is if you are trying to detect user_code
modifying an object passed in, which is something that wouldn't be
addressed by your original complaint about exec (as in, modifying a
global data structure).
Instead of:
if key in return_stuff and return_stuff[key] == context[key]:
Use:
if key in return_stuff and return_stuff[key] is context[key]:
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/27/2010 4:13 AM, Brian Quinlan wrote: On 27 May 2010, at 17:53, Floris Bruynooghe wrote: On Thu, May 27, 2010 at 01:46:07PM +1200, Greg Ewing wrote: On 27/05/10 00:31, Brian Quinlan wrote: You have two semantic choices here: 1. let the interpreter exit with the future still running 2. wait until the future finishes and then exit I'd go for (1). I don't think it's unreasonable to expect a program that wants all its tasks to finish to explicitly wait for that to happen. I'd got for (1) as well, it's no more then reasonable that if you want a result you wait for it. And I dislike libraries doing magic you can't see, I'd prefer if I explicitly had to shut a pool down. I'm glad I'm not alone in preferring (1) tough. Keep in mind that this library magic is consistent with the library magic that the threading module does - unless the user sets Thread.daemon to True, the interpreter does *not* exit until the thread does. Given your rationale, I don't understand from the PEP: shutdown(wait=True) Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to Executor.submit and Executor.map and made after shutdown will raise RuntimeError. If wait is True then the executor will not return until all the pending futures are done executing and the resources associated with the executor have been freed. Can you tell me what is the expected execution time of the following: executor = ThreadPoolExecutor(max_workers=1) executor.submit(lambda: time.sleep(1000)) executor.shutdown(wait=False) sys.exit(0) I believe it's 1000 seconds, which seems to defy my request of shutdown(wait=False) because secretly the Python exit is going to wait anyways. ISTM, it is much easier to get behavior #2 if you have behavior #1, and it would also seem rather trivial to make ThreadPoolExecutor take an optional argument specifying which behavior you want. Your reference implementation does not actually implement the specification given in the PEP, so it's quite impossible to check this myself. There is no wait=True option for shutdown() in the reference implementation, so I can only guess what that implementation might look like. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3148 ready for pronouncement
On 5/26/2010 8:03 PM, Nick Coghlan wrote: On 27/05/10 02:27, Terry Reedy wrote: I am suggesting that if we add a package, we do it right, from the beginning. This is a reasonable point of view, but I wouldn't want to hold up PEP 3148 over it (call it a +0 for the idea in general, but a -1 for linking it to the acceptance of PEP 3148). That sounds backward. How can you justify accepting PEP 3148 into a concurrent namespace without also accepting the demand for such a namespace? What is the contingency if this TBD migration PEP is not accepted, what happens to PEP 3148? After all, there was some complaints about just calling it futures, without putting it in a concurrent namespace. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Enhanced tracker privileges for dangerjim to do triage.
On 4/26/2010 7:24 AM, Antoine Pitrou wrote: I think it is very much in the interest of Python to evolve our processes in order to encourage more core-developers. Evolving means experimenting *and* being willing to change. It is certainly less *effort* to accept the status quo, but with several more committed and enthusiastic (and good) core developers there is an awful lot (more) we could achieve. I certainly agree we should try to attract more good-willed and competent contributors. I also agree with Stephen that, in a project with a non-trivial amount of complexity such as CPython, not having (tracker or commit) privileges is not the real barrier to entry. You have to learn how the software works, how its development works, who are the people working on it, etc. I'd like to respond to Michael's comment about the possibly hundreds of modules in the standard library without a maintainer. My own experience (issue5949) has been positive despite the lack of a dedicated maintainer. When I had my own itch to scratch, nobody stopped me from scratching it.. some people told me when I could scratch it and how they'd like it scratched.. but I wasn't ignored or rejected despite the lack of a maintainer. Thanks to RDM for giving my issue attention. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7b1 and argparse's version action
On 4/18/2010 9:44 PM, Steve Holden wrote: Tobias Herp wrote: Steven Bethard schrieb: On Sun, Apr 18, 2010 at 2:35 PM, Tobias Herp tobias.h...@gmx.de wrote: Steven Bethard schrieb: But I'd really like a consensus about the correct behavior, and so far I have not seen that. Do you take your own poll seriously? When was this ever a democracy? Is consensus superficial? -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3147: PYC Repository Directories
On 1/31/2010 2:04 PM, Raymond Hettinger wrote: On Jan 30, 2010, at 4:00 PM, Barry Warsaw wrote: It does this by allowing many different byte compilation files (.pyc files) to be co-located with the Python source file (.py file). It would be nice if all the compilation files could be tucked into one single zipfile per directory to reduce directory clutter. It has several benefits besides tidiness. It hides the implementation details of when magic numbers get shifted. And it may allow faster start-up times when the zipfile is in the disk cache. On a whim, I implemented a PEP302 loader that cached any important that it could find in sys.path into a zip file. I used running bzr as a startup benchmark, and I did my best to ensure an empty cache by running sync; echo 3 /proc/sys/vm/drop_caches; time bzr. On my particular machine, the real time was at minimum 3.5 seconds without using my ZipFileCacheLoader. With the loader, I found the same was true. The average performance was all over the place (due everything else in the operating system trying to fetch from the disk), and I lack enough data points to reach statistical significance. However, if the .pyr zip file is going to contain many versions of the same module, then the performance impact could be more real, since you would be forced to pull from disk *all* of the versions of a given module. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/29/2010 8:43 AM, Cesare Di Mauro wrote: If you use Mercurial, you can grab a local copy this way: hg clone https://wpython10.wpython2.googlecode.com/hg/ wpython2-wpython10 Wpython is intended to run on any platform where CPython 2.6.4 runs. Cesare, just FYI, your Hg repository has lost the execute bits on some files (namely ./configure and ./Parser/asdl_c.py), so it does not quite build out-of-the-box. I took the liberty of cloning your repo into my laptop's VirtualBox instance of Ubuntu. I ran the default performance tests from the U-S repo, with VirtualBox at highest priority. As a sanity check, I ran it against the U-S trunk. I think the numbers speak for themselves. $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../wpython2-wpython10/python Report on Linux narf-ubuntu 2.6.31-16-generic #53-Ubuntu SMP Tue Dec 8 04:01:29 UTC 2009 i686 Total CPU cores: 1 ### 2to3 ### Min: 21.629352 - 20.893306: 1.0352x faster Avg: 22.245390 - 21.061316: 1.0562x faster Significant (t=4.416683) Stddev: 0.58810 - 0.11618: 5.0620x smaller Timeline: http://tinyurl.com/yawzt5z ### django ### Min: 1.105662 - 1.115090: 1.0085x slower Avg: 1.117930 - 1.131781: 1.0124x slower Significant (t=-11.024923) Stddev: 0.00729 - 0.01023: 1.4027x larger Timeline: http://tinyurl.com/ydzn6e6 ### nbody ### Min: 0.535204 - 0.559320: 1.0451x slower Avg: 0.558861 - 0.572902: 1.0251x slower Significant (t=-7.484374) Stddev: 0.01309 - 0.01344: 1.0272x larger Timeline: http://tinyurl.com/ygjnh5x ### slowpickle ### Min: 0.788558 - 0.757067: 1.0416x faster Avg: 0.799407 - 0.774368: 1.0323x faster Significant (t=12.772246) Stddev: 0.00686 - 0.01836: 2.6759x larger Timeline: http://tinyurl.com/y8g3zjg ### slowspitfire ### Min: 1.200616 - 1.218915: 1.0152x slower Avg: 1.229028 - 1.255978: 1.0219x slower Significant (t=-8.165772) Stddev: 0.02133 - 0.02519: 1.1808x larger Timeline: http://tinyurl.com/y9gg2x5 ### slowunpickle ### Min: 0.355483 - 0.347013: 1.0244x faster Avg: 0.369828 - 0.359714: 1.0281x faster Significant (t=6.817449) Stddev: 0.01008 - 0.01089: 1.0804x larger Timeline: http://tinyurl.com/ybf3qg9 ### spambayes ### Min: 0.316724 - 0.314673: 1.0065x faster Avg: 0.327262 - 0.332370: 1.0156x slower Significant (t=-3.690136) Stddev: 0.00598 - 0.01248: 2.0876x larger Timeline: http://tinyurl.com/ydck59l $ ./perf.py -r -b default \ ../python-2.6.4/python \ ../unladen-swallow/python ### 2to3 ### Min: 24.833552 - 24.433527: 1.0164x faster Avg: 25.241577 - 24.878355: 1.0146x faster Not significant Stddev: 0.39099 - 0.28158: 1.3886x smaller Timeline: http://tinyurl.com/yc7nm79 ### django ### Min: 1.153900 - 0.892072: 1.2935x faster Avg: 1.198777 - 0.926776: 1.2935x faster Significant (t=61.465586) Stddev: 0.03914 - 0.02065: 1.8949x smaller Timeline: http://tinyurl.com/ykm6lnk ### nbody ### Min: 0.541474 - 0.307949: 1.7583x faster Avg: 0.564526 - 0.327311: 1.7247x faster Significant (t=57.615664) Stddev: 0.01784 - 0.03711: 2.0798x larger Timeline: http://tinyurl.com/ylmezw5 ### slowpickle ### Min: 0.832452 - 0.607266: 1.3708x faster Avg: 0.860438 - 0.651645: 1.3204x faster Significant (t=20.779110) Stddev: 0.01559 - 0.09926: 6.3665x larger Timeline: http://tinyurl.com/yaktykw ### slowspitfire ### Min: 1.204681 - 1.038169: 1.1604x faster Avg: 1.236843 - 1.085254: 1.1397x faster Significant (t=20.203736) Stddev: 0.02417 - 0.07103: 2.9391x larger Timeline: http://tinyurl.com/ykgmop5 ### slowunpickle ### Min: 0.374148 - 0.279743: 1.3375x faster Avg: 0.398137 - 0.315630: 1.2614x faster Significant (t=16.069155) Stddev: 0.01333 - 0.04958: 3.7203x larger Timeline: http://tinyurl.com/y9b5rza ### spambayes ### Min: 0.330391 - 0.302988: 1.0904x faster Avg: 0.349153 - 0.394819: 1.1308x slower Not significant Stddev: 0.01158 - 0.35049: 30.2739x larger Timeline: http://tinyurl.com/ylq8sef -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch to make list.pop(0) work in O(1) time
On 1/28/2010 11:57 PM, Steve Howell wrote: --- On Thu, 1/28/10, Josiah Carlson josiah.carl...@gmail.com wrote: [...] in the decade+ that I've been using Python and needed an ordered sequence; lists were the right solution 99% of the time [...] What do you think of LISP, and car in particular (apart from the stupidly cryptic name)? A LISP list/pair has nothing to do with a Python list. The list of LISP is a *singly-linked* list. You cannot O(1) index a LISP list. A Python list is more equivalent to a LISP vector, which car does not work with; in fact, there is not even a pop operator -- all size changes of a vector O(n) unless the implementation is playing games (like the one you are proposing for the start and the one Python already uses for the end of a list). (And with this, clearly uninformed reply by you, I am now ignoring your trolling.) -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed downstream change to site.py in Fedora (sys.defaultencoding)
On 1/22/2010 2:04 PM, M.-A. Lemburg wrote: Karen Tracey wrote: On Fri, Jan 22, 2010 at 7:38 AM, Michael Foord fuzzy...@voidspace.org.ukwrote: The encoding I'm talking about is the encoding that Python uses to decode a file (or encode a string) when you do the following in Python 3: text = open(filename).read() open(filename, 'w').write(some_string) It isn't the default encoding (always utf-8 by default in Python 3 apparently), it isn't the file system encoding which is the system encoding used for file names. What is the correct terminology for this platform dependent encoding that Python uses here? The doc here: http://docs.python.org/3.1/library/functions.html?highlight=open#open just calls it default encoding and clarifies that is whatever locale.getpreferredencoding() returns. ... which is a pretty poor guess, since the locale setting is usually set to what the user wants to see in user interfaces, not what the application want to use as file content. [...] Applications should always provide a well-defined encoding for use with files - they know best what encoding to use and they also know best when to apply guessing and which set of encodings to use as basis for such guessing. This all begs the question: why is there a default? and why is the default a guess? I have to admit that I was completely oblivious to this potential pitfall, and mostly that's because in the most common case, I am working with ASCII files. It's just serendipity that most systems specify (if not require) the locale encoding be an ASCII superset. locale.getpreferredencoding([do_setlocale]) Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess. I already know that this suggestion will not get any following because, for most people, it just works. However: In the face of ambiguity, refuse the temptation to guess. Would it really be that unfortunate to force everyone to reconsider what they are doing when they open() files? -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On 1/21/2010 1:09 PM, Hanno Schlichting wrote: - Who holds the copyright, is there a foundation or is there a risk of the project getting into trouble because of copyright issues? - What licence is the code and the tools under and what affect does that have on the code generated by the JIT compiler? http://llvm.org/docs/DeveloperPolicy.html#clp We intend to keep LLVM perpetually open source and to use a liberal open source license. The current license is the University of llinois/NCSA Open Source License[1], which boils down to this: * You can freely distribute LLVM. * You must retain the copyright notice if you redistribute LLVM. * Binaries derived from LLVM must reproduce the copyright notice (e.g. in an included readme file). * You can't use our names to promote your LLVM derived products. * There's no warranty on LLVM at all. We believe this fosters the widest adoption of LLVM because it allows commercial products to be derived from LLVM with few restrictions and without a requirement for making any derived works also open source (i.e. LLVM's license is not a copyleft license like the GPL). We suggest that you read the License[1] if further clarification is needed. We have no plans to change the license of LLVM. If you have questions or comments about the license, please contact the LLVM Oversight Group[2]. [1] http://www.opensource.org/licenses/UoI-NCSA.php [2] llvm-oversi...@cs.uiuc.edu -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Download Page - AMD64
On 1/13/2010 9:04 AM, Nick Coghlan wrote: As Windows doesn't run on non-x86 architectures, their naming is generally just Windows Whatever (32 bit) and Windows Whatever (64 bit). That is not correct. There are IA-64 versions of Window Server. From [1]: Backward compatibility is a key point differentiating Itanium from x86 and x64 architectures. So to echo what Michael said, the Microsoft nomenclature is x64 regardless of yours and Martin's objections to that name. Nobody who uses Windows would be confused by x64 since that is *the* Microsoft naming scheme. And, anyone using IA64 will expect to see IA64 and not x64, and furthermore anyone using IA64 is likely aware of what processor they are using (being as there are only Windows Server editions for IA64) and the difference between IA64 and x64. I don't think an end-user facing page is a great place for pedantic and wordy defenses of defying the Microsoft-blessed naming scheme. Linux, on the other hand, can run on multiple 64 bit architectures, so being more specific and using the official AMD64 nomenclature makes sense. This has no relevance to the conversation since there are no Linux binaries being distributed. The conversation on the expectations of Windows end-users, who are the target of the download links. [1] http://www.microsoft.com/servers/64bit/itanium/overview.mspx -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing PEP 345 : Metadata for Python Software Packages 1.2
On 12/28/2009 5:42 AM, Martin v. Löwis wrote: So specifying 2.5 would be a short-hand for *what*? 2.5 would be a shorthand for 2.5.x. So, equivalent to : =2.5.0, 2.6.0 Ok, so it's not a shorthand for a single operator anymore, but for a more complex term. Fine with me. 2.5.0 would be the notation required to describe this specific micro version. I think it would be a shorthand for =2.5.0, 2.5.1, right? Actually, the logical extension is that 2.5.0 is shorthand for =2.5.0.0, 2.5.1.0, I believe. PEP 386 versions can have an indefinite number of extradecimal versions. Pedantically, -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASED] Python 2.7 alpha 1
Benjamin Peterson wrote: On behalf of the Python development team, I'm pleased to announce the first alpha release of Python 2.7. Was there a call for bugs that should be reviewed for this release? I once again will prod for attention to this show-stopping bug for using IMAP-SSL: http://bugs.python.org/issue5949 Title: IMAP4_SSL spin because of SSLSocket.suppress_ragged_eofs -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3003 - Python Language Moratorium
Guido van Rossum wrote: On Sun, Nov 8, 2009 at 3:06 PM, Steven D'Aprano st...@pearwood.info wrote: No new language features in odd-numbered point releases (3.3, 3.5, ...). Even-numbered point releases (3.4, 3.6, ...) may include new language features provided they meet the usual standards for new features. Oh no, not the eve/odd numbering scheme. Nobody will be able to remember which is which. In this case, does it really matter? Usually the confusion is with stable/unstable being even/odd or odd/even, but in this case the options are stable/stable (it just happens that certain pairwise versions are related). -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL behaviour under Windows
Antoine Pitrou wrote: Could you try ccbench (*) under Windows? The only Windows system I have here is a qemu virtual machine and it wouldn't be very realistic to do concurrency measurements on it. (*) http://svn.python.org/view/sandbox/trunk/ccbench/ I don't really know how this test works, so I won't claim to understand the results either. However, here you go: C:\systeminfo OS Name: Microsoft Windows XP Professional OS Version:5.1.2600 Service Pack 3 Build 2600 C:\c:\Python26\python.exe Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32 C:\start /B /HIGH c:\Python26\python.exe c:\ccbench.py --- Throughput --- Pi calculation (Python) threads=1: 377 iterations/s. threads=2: 376 ( 99 %) threads=3: 380 ( 100 %) threads=4: 376 ( 99 %) regular expression (C) threads=1: 222 iterations/s. threads=2: 213 ( 95 %) threads=3: 223 ( 100 %) threads=4: 218 ( 97 %) bz2 compression (C) threads=1: 324 iterations/s. threads=2: 324 ( 99 %) threads=3: 327 ( 100 %) threads=4: 324 ( 100 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 0 ms. (std dev: 0 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL behaviour under Windows
Antoine Pitrou wrote: Interesting results. I wonder what they would be like on a multi-core machine. The GIL seems to behave perfectly on your setup (no performance degradation due to concurrency, and zero latencies). You are correct, my machine is a single-core system. I don't have any multi-core systems around to test it on, I'm still in the stone age. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] time.clock() on windows
Curt Hagenlocher wrote: But it makes more sense to understand why someone chose to implement time.clock() on Windows the way they did -- this seems very broken to me, and I think it should be changed. Some SVN detective work takes this to all the way back to r7713 (1997-04-02). The original implementation checked by Guido and attributed to Mark Hammond. So, we should ask Mark why he did that. Can anyone honestly use it, as it is, without already having normalize it across platforms themselves? I don't know how much of an impact it is, but the current implementation of clock() does not require argument parsing, so the proposal to add a absolute boolean-flag argument is perhaps bad. This is generally a function used for performance timing and that proposal adds some amount of latency to the query. The proposal to add a clockbase() function is perhaps better because of this, you need only call it once, and you can cache the result for the life of your process. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP about sys.implementation and implementation specific user site directory
Christian Heimes wrote: id (required): lower case identifier, for example cpython, ironpython, jython, pypy name (required): mixed case name of the implementation, for example CPython, IronPython, Jython, PyPy Why both? Is it not true that the following is guaranteed? sys.implementation.id == sys.implementation.name.lower() Furthermore, why is a lower-case-only constant superior to a mixed-case identifier? I wonder this especially given that platform.python_implementation() already returns mixed-case constants. Why not include the version of the language? platform (required): platform or language of the implementation, for example C, .NET, Java What should someone like PyPy say here? Do you have a use-case in mind? What is the relevance of this field? runtime (required): lower case runtime library of the implementation, for example libc, msvcrt90, java6, .net Same questions as above. Is the java-runtime not dependent on libc? $ ldd /opt/sun-jdk-1.6.0.15/jre/bin/java | grep libc libc.so.6 = /lib/libc.so.6 (0xb7d7) compiler (required): verbose name of the compiler, for example GCC 4.3.3, Java HotSpot(TM) Client VM (Apple Inc.), Mono JIT compiler version 2.0.1 I think other commenters have already asked what exactly is considered the compiler? This is especially interesting if you consider PyPy, as well. What is the item being compiled that you are interested in? What is the use-case for this info? It could vary based on module: some modules could've been built with a different compiled or even compiled to bytecode by a different implementation or etc. It seems like (platform, runtime, compiler) are guesses at what *might* be useful. But, in the absence of use-cases they are not useful, perhaps misguided, and create a maintenance burden for the future. A lot of this info is available via the platform module. And the only reason given to not use the platform module was to avoid adding dependencies for site.py. I don't see that as an open invitation to adding other fields that are already available via the platform module. Until they are critical to run-time performance, why not wait to add these extra things? The only thing that has been indicated as needed is the identifier for the python implementation. sys.vm or sys.implementation may very well fully support the use cases given merely by being a string. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] a new setuptools release?
P.J. Eby wrote: At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: suggest nobody hold their breath waiting for setuptools 0.7. I've never suggested or implied otherwise. But, if you like Distribute so much, why not just add it directly to the stdlib? ;-) There are many wins to be had from such a move, not the least of which is that it allows something to go into the stdlib that isn't (branding-wise) associated with me or setuptools, and lets it become owned/supported by an even wider community. I think the biggest problem here is that the brand (setuptools) is so ingrained in the world that someone releasing something setuptools-but-not-setuptools (Distribute) is at a serious disadvantage. Your insistence on retaining the name setuptools for yourself and your vapor-ware only harms the community at-large. Even experienced developers are unaware of Distribute's existence.. I was entirely unaware until yourself and PJE got in a bickering exchange on Python-Dev this past month. The extremely high visibility of your stale package compared to their non-stale package is quite unfortunate. You are free to say, that is their problem, but you are not free to say, it is not my fault. AFAIK, the only reason they've had multiple releases of it is to address the issues with its hijacking of setuptools; in a stdlib version all that stuff could be dropped. Otherwise, it'd already be mature. IOW, you acknowledge that your own position and requiring them to tolerate the parallel existence (in the world) of setuptools harms Distribute. I fail to see how this relates to being in the stdlib. As long as it is not called setuptools and packages in the world say import setuptools, then there are conflicts they will be forced to managed. And moreover, as long as a stale, perhaps buggy version of setuptools is the compatibility model they must emulate, they will have a hard time coexisting. Less political bickering, and the some of the technical results I hoped for all along are achieved. Yay, open source. And yet, political bickering seems to be all you are good for in this case. /flame -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.4rc1
Barry Warsaw wrote: 2.6.4 final is planned for 18-October. Barry, I suspect this release is primarily to quench the problems with distutils, but.. http://bugs.python.org/issue5949 doesn't seem to have been addressed by you. And this seems like it would be another unfortunate loss of an opportunity. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Barry Warsaw wrote: If not, I'll try to spend some time over the next few days looking at outstanding bugs, and marking release blockers, etc. I'd like to draw attention to: http://bugs.python.org/issue5949 Which is a regression of imaplib.py introduced in r57680. Ever since I switched to python 2.6 on my box, I have had issues with getmail[1] getting stuck in an infinite loop swallowing memory (note: only applies to IMAP SSL connections). While this code is present in older versions of python, it seems to have become a problem recently (2009-05-06 is the earliest report on the issue) perhaps due to a version bump of OpenSSL? I never noticed the problem in python2.5 even though the code is unchanged. Nevertheless, the code is clearly wrong and should be corrected. I would appreciate this bug being resolved before the next release as it effects me on a daily basis. I have submitted a patch, which reflects my local solution. -Scott [1] http://pyropus.ca/software/getmail/ -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Scott Dial wrote: While this code is present in older versions of python, it seems to have become a problem recently (2009-05-06 is the earliest report on the issue) perhaps due to a version bump of OpenSSL? I never noticed the problem in python2.5 even though the code is unchanged. To answer my own question, this was introduced in r64578 in ssl.py by the addition of the suppress_ragged_eofs feature (seems to originate with issue1251). While it is a change in the behavior of the ssl module, the change falls in line with other file-like objects and their handling of EOF, so the bug still falls to imaplib. In other words, this bug appeared in 2.6 and 3.0, and is present in all subsequent versions. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.6.3
Nick Coghlan wrote: Scott Dial wrote: I would appreciate this bug being resolved before the next release as it effects me on a daily basis. I have submitted a patch, which reflects my local solution. Unfortunately, it's almost certainly too late to get this into 2.6.3. It really needed to be brought up back when Barry called for identification of significant 2.6 bugs and patches rather than after the RC had already been released. I understand. I didn't figure out the bug until after rc1 landed. It was only happening spuriously with my mail server, never when I manually tried to invoke the problem. I was forced to let my cronjob run until the kernel killed getmail for OOM, giving me the trace shown in the issue. It is impossible to break anything with this patch, as no program could proceed. The 3-line patch merely converts it back into the exception that was originally raised prior to 2.6, so it's not a new behavior in that respect either. I wouldn't have anticipated this much resistance to removing an infinite loop from the standard library. I could also for this patch from the angle that it allows a remote host the ability to execute a denial-of-service attack (even by accident!) since the infinite loop appends an empty string to a list on every loop, taking CPU time and memory with it. Allow me to be naive for a moment and say, is this not the point of rc1 but to catch bugs that should not be in the final? Of course, it's Barry's call. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Martin v. Löwis wrote:
Martin v. Löwis martin at v.loewis.de writes:
Could you explain what benefit there is for allowing the user to create
network objects that don't represent networks? Is there a use-case
where these networks-that-aren't-networks are something other than a
typo? Under what circumstances would I want to specify a network as
192.168.1.1/24 instead of 192.168.1.0/24?
[...]
So Python code has to make the computation, and it seems most natural
that the IP library is the piece of code that is able to compute a
network out of that input.
The thing is, it doesn't create a network, it creates a hybrid network plus
host which retains knowledge about the original host (that is, 192.168.1.1
rather than simply 192.168.1.0, if you enter 192.168.1.1/24).
That's what the OP meant with networks-that-aren't-networks, and that's
what
people are objecting to.
That's not the question that was asked, though - the question asked
was Under what circumstances would I want to specify I hope
most people agree that it is desirable to be able to specify a network
not just by its network address.
I can't help but feel that you and many others jumping on Antoine et al.
have skipped reading or only skimmed this topic of discussion.
Antoine Pitrou wrote:
We just need a factory function which returns a tuple after parsing:
addr, net = parse_address_with_mask('192.168.1.1/24')
addr == IPv4Address('192.168.1.1')
True
net == IPv4Network('192.168.1.0/24')
True
I still back this suggestion. I would also support a loose=True flag
for the IPNetwork constructor, which would allow host bits to be non-zero.
However, I will *never* support a proposal that includes retaining the
host address. I believe I am not alone in that. However, Peter has made
it quite clear that he will *never* change that about the IPNetwork classes.
I would be quite glad to not have ipaddr included in the stdlib. As I
can only imagine code that uses it being quite confusing to maintain
(e.g., wait is that actually a network or is that someone using
IPNetwork to be an IPAddressWithMask?).
-1.
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote: On Wed, Sep 16, 2009 at 8:21 PM, Andrew McNamara andr...@object-craft.com.au wrote: I think we're in a painful middle ground now - we should either go back to the idea of a single class (per protocol), or make the distinctions clear (networks are containers and addresses are singletons). Personally, I think I would be happy with a single class (but I suspect that's just my laziness speaking). However, I think the structure and discipline of three classes (per protocol) may actually make the concepts easier to understand for non-experts. I think this is where we disagree. I don't think the added complexity does make it any easier to understand. I argue that we're not actually adding any complexity: yes, we add a class (per protocol), but we then merely relocate functionality to clarify the intended use of the classes. And I argue the moving this functionality to new classes (and adding new restrictions to existing classes) doesn't buy anything in the way of overall functionality of the module or a developer's ability to comprehend intended uses. Speaking as the originator of this thread of discourse, the lack of a 3rd class was exactly the source of my confusion and my inability to communicate my confusion to everyone. I clearly did not understand the intended uses of the IPNetwork type because it was capable of playing two roles that are decidedly different conceptually. So, I must respectfully disagree with you. You have an address + netmask. ergo, you have a Network object. In a common use case, however, this instance will not represent a network at all, but an address. It will have container-like behaviour, but it should not (this is a property of networks, not addresses). So the instance will be misnamed and have behaviours that are, at best, misleading. This is exactly the confusion and duality that I struggled with. This isn't about shortcuts, but about correctness... having the Network object represent a network, and having Address objects represent end-points, and having errors discovered as early as possible. Then what I don't see is the purpose of your network-only-network-object. essentially identical functionality can be obtained with the module as is w/o the added complexity of new classes. The purpose is to avoid conflating IPNetwork with an IPAddress that has a mask. If the IPNetwork didn't accept a non-zero host and instead required a developer to use a helper to construct a IPNetwork with a proper address, then there would be less confusion about what exactly a IPNetwork is meant to represent. As it stands, it's purposes is muddled by accepting host addresses too. I am rather indifferent whether there needs to be a IPAddressWithMask type. If that is needed, then it is rather easy to create a type that does that. And, if it is a common pattern, then it could be added to the module later in life. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
R. David Murray wrote:
On Tue, 15 Sep 2009 at 14:28, Antoine Pitrou wrote:
Andrew McNamara andrewm at object-craft.com.au writes:
ipaddr.IPv4Network('192.168.1.1/16').network
IPv4Address('192.168.0.0')
Er, does this mean that taking the `network` attribute from a network
object
actually gives an address object (not a network)?
It looks horribly misleading to me.
That was my opinion, too. I've always called that number the 'zero'
of the network, but other people said that it is usually called the
'network'.
I also find it odd to see a 'network' with an IP address of 192.168.1.1.
That looks like a host-address-plus-netmask to me, not a network
address.
It just happened that I needed a tool today to calculate the gateway IP
for an interface, and I took it as an opportunity to try out this ipaddr
module. I'll attempt to relate my experience below...
I have to concur with the opinions above. I was very confused by the
following error:
addr = ipaddr.IPAddress(10.1.2.3/255.255.240.0)
...
ipaddr.IPAddressIPValidationError: '98.223.189.24/255.255.240.0' is not
a valid address (hint, it's probably a network)
Because, it *is* a address of a host on a network. I gave in and said:
net = ipaddr.IPNetwork(10.1.2.3/255.255.240.0)
But then, I was dumbfounded as to how I could get the gateway IP from
this IPNetwork object. It took me a while to figure out that you can
iterate over IPNetwork instances:
gateway = net[1]
I was then confused, because:
print(type(gateway))
class 'ipaddr.IPv4Address'
Which sorta blew my mind.. I fully expected to receive an IPNetwork back
from that operation. It is unclear to me why the network information
gets chucked by that operation. I foresee having to work around that in
real applications by doing something obnoxious like:
actual_gateway = ipaddr.IPNetwork(%s/%s % (gateway, addr.netmask))
But since I only actually needed the IP address for the gateway, it
fulfilled my needs.
In the end, I found the names IPNetwork/IPAddress and their
instantiations confusing. ISTM that IPNetwork is overloaded and plays
two roles of being an IPNetwork and being an IPAddressWithNetwork. And
finally, it's unclear to me why iterating over a IPNetwork would not
produce a sequence of IPNetwork(/IPAddressWithNetwork) objects.
ISTM that if I started with an IPNetwork object, the API should always
return IPNetwork objects. If I want just an IPAddress object, then I can
always fetch the ip attribute to get that. Perhaps there is some
compelling conceptual argument as to why this is not correct, but it
seems like the API destroys information needlessly.
Just my opinion..
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote:
On Tue, Sep 15, 2009 at 10:16 AM, Scott Dial
scott+python-...@scottdial.com wrote:
In the end, I found the names IPNetwork/IPAddress and their
instantiations confusing. ISTM that IPNetwork is overloaded and plays
two roles of being an IPNetwork and being an IPAddressWithNetwork. And
finally, it's unclear to me why iterating over a IPNetwork would not
produce a sequence of IPNetwork(/IPAddressWithNetwork) objects.
What you're describing, at least for the networks, is basically what
ipaddr was. It seemed, after much heated discussion, that the
community felt that
ipaddr.IPv4Network('1.1.1.0/24')[0]
was actually an individual address. ie, each of the addresses in
1.1.1.0/24 are individual addresses, rather than networks with /32
prefix lengths.
For clarity, I am going to call the current design A:
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Address('1.1.1.0')
And what itt sounds like what you are describing as the old behavior is
this (design B):
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Network('1.1.1.0/32')
Which is different than what I was trying to describe. I expected
(design C):
ipaddr.IPv4Network('1.1.1.0/24')[0] == ipaddr.IPv4Network('1.1.1.0/24')
My summarization of these designs would be that B is wrong. And that
A is better than B, certainly. At least A is not saying something
untrue. However, C would be truthful and retains a superset of
information that A provides (the ip attribute of IPNetwork).
In other words, I don't see why obtaining a host address would *not*
retain the hostmask from the network it was obtained from. I am not
disagreeing with it being an individual address. I am disagreeing that
IPNetwork itself already does represent individual addresses (hence my
aliasing it with IPAddressWithNetwork). And wrt, the logical return
would be another IPAddressWithNetwork retaining the same mask.
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Peter Moody wrote:
but that's what's being suggested here for networks.
ipaddr.IPv4Network('1.1.1.1/24')[0][1][2][3]...
This example here solidifies my support of RDM's suggestion of there
being 3 types:
IPv4Address
IPv4AddressWithNetwork (or as he called it: IPv4AddressWithMask)
IPv4Network
The primary difference between IPv4Network and IPv4AddressWithNetwork
would be that IPv4AddressWithNetwork would not support
iteration/indexing. The example above makes no sense logically and my
original suggestion unknowingly opened that can of worms. If indexing a
IPv4Network returned IPv4AddressWithNetwork, then that would remove that
oddity.
This would also solve the weirdness that Stephen brought up in another
branch of this discussion:
Stephen J. Turnbull wrote:
Scott Dial writes:
ipaddr.IPv4Network('1.1.1.0/24')[0] ==
ipaddr.IPv4Network('1.1.1.0/24')
So foo returns True?
def foo():
a = ipaddr.IPv4Network('1.1.1.0/24')
return a[0] == a
That seems ... weird. Maybe you committed a typo?
The root of the weirdness is that my proposition would appear to make
IPv4Network play double-duty. However, it already does! That you can
instantiate a IPv4Network object with a non-zero host is a clue, and
that it is apparently the only way to input a host with a mask is the
second clue.
If I have an IP (10.2.3.4) and I know the netmask (say, 255.255.0.0),
then how do I get the network that it is on? As it stands, the only way
I see to do this is to do:
net = ipaddr.IPv4Network(10.2.3.4/255.255.0.0)
Which is strange, because I didn't input a *network*, I inputted an
*address* on a network. Moreover, it's strange because:
net[0] == net.ip
False
If I needed that identity to hold true, then I have to do:
net = ipaddr.IPv4Network(%s/%s % (net[0], net.netmask)
Although it is unclear why a network has an ip attribute. As far as
I am concerned, IPv4Network objects should *only* have a net.network.
Hopefully at this point, I have made the case that IPv4Network already
is playing double-duty as a IPv4Network and IPv4AddressWithNetwork. And
that the API would be well-served by splitting that role and that it
would be considerably less confusing.
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
R. David Murray wrote:
On Tue, 15 Sep 2009 at 21:58, Antoine Pitrou wrote:
Le mardi 15 septembre 2009 à 15:48 -0400, R. David Murray a écrit :
However, I do not think
that the proposed API should accept, eg, IPv4Network('192.168.1.1/24')
as valid. That's just too confusing and error prone.
Indeed, it should throw some kind of ValueError instead.
Peter, what do you think?
This change would have to be dependent on Antoine's suggestion of:
addr, net = parse_address_with_mask('192.168.1.1/24')
addr == IPv4Address('192.168.1.1')
True
net == IPv4Network('192.168.1.0/24')
True
Otherwise, I am ok with this change too. It resloves the weird duality
of IPv4Network. And as RDM says, anyone who wants the
IPv4AddressWithNetwork functionality can just role it themselves. At the
moment of iteration, you have access to what network it is and can build
your own IPv4AddressWithNetwork objects.
net = IPv4Network('10.0.0.0/24')
netaddrs = []
for addr in net:
netaddrs.append(IPv4AddressWithNetwork(addr, net))
I guess I am ok with this. It seems sub-optimal (why not just return a
IPv4AddressWithNetwork to begin with?) but I suppose it is no less
efficient since the same objects would be constructed.
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3144 review.
Sebastian Rittau wrote: On Tue, Sep 15, 2009 at 01:16:06PM -0400, Scott Dial wrote: net = ipaddr.IPNetwork(10.1.2.3/255.255.240.0) But then, I was dumbfounded as to how I could get the gateway IP from this IPNetwork object. Well, you can't. There is no way to determine a gateway, without querying the network topology. This is clearly outside the scope of this module. The only two known host addresses of a network are the network address (10.1.0.0 in the example you gave) and the broadcast address (10.1.15.255). I was not writing a general utility for the world to use. On *my* network it is always the [1]th address used for the gateway. But, I agree in general that is not true. Which sorta blew my mind.. I fully expected to receive an IPNetwork back from that operation. It is unclear to me why the network information gets chucked by that operation. This makes perfect sense to me. An IP network consists of a list of IP addresses. Returning /32 networks seems kind of pointless to me. As I clarified later, I expected it to still be a /24 network, not a /32. /32 makes no sense. But a gateway is not an IP address plus hostmask. A gateway is just a single IP address. What is the use of adding a hostmask to the gateway IP address or some other IP address inside the network? My argument was from a destruction of information standpoint. Why is the network chucked away? Wouldn't it be handy to be able to inspected addr.network to see what network it was on? Perhaps because you are building routing rules and need to actually know that. As it is, you will just have to build your own IPAddressWithNetwork objects. Hopefully I have covered all of you inquiries, but I feel like that a lot of this is covered by many other replies in this thread. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] how to debug httplib slowness
Chris Withers wrote:
- Apache in this instance responds with HTTP 1.1, even though the wget
request was 1.0, is that allowed?
- Apache responds with a chunked response only to httplib. Why is that?
I find it very confusing that you say Apache since your really want to
say Coyote which is to say Tomcat.
http11processor.j...@1547:
if (entityBody http11 keepAlive) {
outputBuffer.addActiveFilter
(outputFilters[Constants.CHUNKED_FILTER]);
contentDelimitation = true;
headers.addValue(
Constants.TRANSFERENCODING).setString(Constants.CHUNKED);
} else {
outputBuffer.addActiveFilter
(outputFilters[Constants.IDENTITY_FILTER]);
}
So, as Oleg said, it's because httplib talks HTTP/1.1 and wget talks
HTTP/1.0. All HTTP/1.1 client must support chunked transfer-encoding,
and apparently Tomcat/Coyote defaults to that unless it is either an
empty message, not a HTTP/1.1 client, or the request is not to be kept
alive (Connection: close or no more keep-alive slots on the server).
As Simon said, changing this to do ''.join(chunks) is really the best
first step to take.
-Scott
--
Scott Dial
sc...@scottdial.com
scod...@cs.indiana.edu
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Raymond Hettinger wrote: I was just wondering if a bytecode for a superinstruction of the common sequence: 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE might be worth it. [Collin Winter] I doubt it. You'd save a bit of stack manipulation, but since this will only appear at the end of a function, I'd be skeptical that this would make any macrobenchmarks (statistically) significantly faster. I concur with Collin. And since it appears only at the end of a function, the optimization doesn't help inner-loops in a function (where most of the time usually spent). I fail to understand this crude logic. How often is the inner-loop really going to solely call C code? Any call to Python in an inner-loop is going to suffer this penalty on the order of the number of loop iterations)? -Scott -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
Raymond Hettinger wrote: I was just wondering if a bytecode for a superinstruction of the common sequence: 6 POP_TOP 7 LOAD_CONST 0 (None) 10 RETURN_VALUE might be worth it. [Collin Winter] I doubt it. You'd save a bit of stack manipulation, but since this will only appear at the end of a function, I'd be skeptical that this would make any macrobenchmarks (statistically) significantly faster. I concur with Collin. And since it appears only at the end of a function, the optimization doesn't help inner-loops in a function (where most of the time usually spent). I fail to understand this crude logic. How often is the inner-loop really going to solely call C code? Any call to Python in an inner-loop is going to suffer this penalty on the order of the number of loop iterations)? -Scott -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] functools.compose to chain functions together
Greg Ewing wrote:
Jason R. Coombs wrote:
I had a use case that was compelling enough that I thought there
should be something in functools to do what I wanted.
I think this is one of those things that a small minority of
people would use frequently, but everyone else would use
very rarely or never. The decision on whether to include
something in the stdlib needs to be based on the wider
picture.
In this case, it's trivial to write your own if you want
it. As they say, not every one-line function needs to
be in the stdlib.
I have never found these arguments compelling. They are obviously not
true (e.g., itertools.compress()[1] added in 2.7/3.1), and so what I
really hear is: I don't like it and I outrank you.
I can't help invoke part of PEP309's justification for
functools.partial()[2]:
I agree that lambda is usually good enough, just not always. And I want
the possibility of useful introspection and subclassing.
The same reasoning would seem to apply here. In the OP's example, the
meta-decorator becomes opaque due to the use of a lambda. If one could
introspect a compose(), then introspection tools could actually know the
set of decorators being applied. As it is, the preferred method of
using a lambda actually makes it quite hard to know anything.
class compose():
def __init__(self, *funcs):
if not funcs:
raise ValueError(funcs)
self.funcs = funcs
def __call__(self, *args, **kwargs):
v = self.funcs[-1](*args, **kwargs)
for func in reversed(self.funcs[:-1]):
v = func(v)
return v
meta = functools.compose(decorator_a, decorator_b)
print meta.funcs
meta = lambda f: decorator_a(decorator_b(f))
# impossible, short of disassembling the lambda
-Scott
[1] http://docs.python.org/3.1/library/itertools.html#itertools.compress
def compress(data, selectors):
# compress('ABCDEF', [1,0,1,0,1,1]) -- A C E F
return (d for d, s in zip(data, selectors) if s)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
Tarek Ziadé wrote: On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbullstep...@xemacs.org wrote: Another general principle: even in the draft PEP, say is, not will be. Ok I'll fix that. That's a French stuff : in french, will be isn't speculative at all. I don't think will be is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, X will be done. If X is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do X. In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of will be in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tuples and underorderable types
Raymond Hettinger wrote: Would it make sense to provide a default ordering whenever the types are the same? def object.__lt__(self, other): if type(self) == type(other): return id(self) id(other) raise TypeError No. This only makes it more difficult for someone wanting to behave smartly with incomparable types. I can easily imagine someone wanting incomparable objects to be treated as equal wrt. sorting. I am thinking especially with respect to keeping the sort stable. I think many developers would be surprised to find, a = tasks = [(10, lambda: 0), (20, lambda: 1), (10, lambda: 2)] tasks.sort() assert tasks[0][1]() == 0 , is not guaranteed. Moreover, I fail to see your point in general as a bug if you accept that there is not all objects can be total ordered. We shouldn't be patching the object base class because of legacy code that relied on sorting tuples; this code should be updated to either use a key function. -Scott -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 380 (yield from a subgenerator) comments
P.J. Eby wrote: One remaining quirk or missing piece: ISTM there needs to be a way to extract the return value without using a yield-from statement. I mean, you could write a utility function like: def unyield(geniter): try: while 1: geniter.next() except GeneratorReturn as v: return v.value My first thought was to ask why it was not equivalent to say: x = yield g x = yield from g This would seem like a more obvious lack of parallelism to pick on wrt. return values. This unyield() operation seems contrived. Never before have you been able to write a generator that returns a value, why would these suddenly become common practice? The only place a return value seems useful is when refactoring a generator and you need to mend having loss of a shared scope. What other use is there for a return value? It would seem unfortunate for it to be considered a runtime error since this would prevent sharing a generator amongst yield from and non-yield from use cases. Although, it would be trivial to do: class K: ... def _f(): yield 1 return 2 # used internally def f() # squelch the runtime error yield from self._f() As Greg has said a number of times, we allow functions to return values with them silently being ignored all the time. -- Scott Dial sc...@scottdial.com scod...@cs.indiana.edu ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
