Re: [Python-Dev] collections.sortedtree
Chris Angelico writes: Don't forget, of course, that there is a middle ground. Something that's really REALLY awesome on PyPI but isn't in the stdlib might be packaged by various Linux distros. Oh, agreed, and any organization that cares that much will already have the RHEL or Ubuntu LTS (etc, those are the ones my buddies use) contract. CPython is not the only certification that is trusted (to whatever degree) by users of CPython. But forget Debian and Gentoo at least -- all it takes get in to those distros is a license (free) and a maintainer (at application time). And they'll even wink at the license given sufficient user demand. Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 28 March 2014 05:09, Josiah Carlson josiah.carl...@gmail.com wrote: So yeah. Someone want to make a decision? Tell me to write the docs, I will. Tell me to go take a long walk off a short pier, I'll thank you for your time and leave you alone. I had a need for this a few years ago. It's messy to do on Windows (ctypes callouts to PeekNamedPipe to check if you can read from the process without blocking). So I would like to see a recipe for this, (even if it's likely to be another few years before I ever need it again :-)). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
2014-03-28 2:16 GMT+01:00 Josiah Carlson josiah.carl...@gmail.com:
def do_login(...):
proc = subprocess.Popen(...)
current = proc.recv(timeout=5)
last_line = current.rstrip().rpartition('\n')[-1]
if last_line.endswith('login:'):
proc.send(username)
if proc.readline(timeout=5).rstrip().endswith('password:'):
proc.send(password)
if 'welcome' in proc.recv(timeout=5).lower():
return proc
proc.kill()
I don't understand this example. How is it asynchronous? It looks
like blocking calls. In my definition, asynchronous means that you can
call this function twice on two processes, and they will run in
parallel.
Using greenlet/eventlet, you can write code which looks blocking, but
runs asynchronously. But I don't think that you are using greenlet or
eventlet here.
I take a look at the implementation:
http://code.google.com/p/subprocdev/source/browse/subprocess.py
It doesn't look portable. On Windows, WriteFile() is used. This
function is blocking, or I missed something huge :-) It's much better
if a PEP is portable. Adding time.monotonic() only to Linux would make
the PEP 418 much shorter (4 sentences instead of 10 pages? :-))!
The implementation doesn't look reliable:
def get_conn_maxsize(self, which, maxsize):
# Not 100% certain if I get how this works yet.
if maxsize is None:
maxsize = 1024
...
This constant 1024 looks arbitrary. On UNIX, a write into a pipe may
block with less bytes (512 bytes).
asyncio has a completly different design. On Windows, it uses
overlapped operations with IOCP event loop. Such operation can be
cancelled. Windows cares of the buffering. On UNIX, non-blocking mode
is used with select() (or something faster like epoll) and asyncio
retries to write more data when the pipe (or any file descriptor used
for process stdin/stdoud/stderr) becomes ready (for reading/writing).
asyncio design is more reliable and portable.
I don't see how you can implement asynchronous communication with a
subprocess without the complex machinery of an event loop.
The API above can be very awkward (as shown :P ), but that's okay. From
those building blocks a (minimally) enterprising user would add
functionality to suit their needs. The existing subprocess module only
offers two methods for *any* amount of communication over pipes with the
subprocess: check_output() and communicate(), only the latter of which
supports sending data (once, limited by system-level pipe buffer lengths).
As I wrote, it's complex to handle non-blocking file descriptors. You
have to catch EWOULDBLOCK and retries later when the file descriptor
becomes ready. The main thread has to watch for such event on the file
descriptor, or you need a dedicated thread. By the way,
subprocess.communicate() is currently implemented using threads on
Windows.
Neither allow for nontrivial interactions from a single subprocess.Popen()
invocation. The purpose was to be able to communicate in a bidirectional
manner with a subprocess without blocking, or practically speaking, blocking
with a timeout. That's where the async term comes from.
I call this non-blocking functions, not async functions.
It's quite simple to check if a read will block on not on UNIX. It's
more complex to implement it on Windows. And even more complex to
handle to add a buffer to write().
Your next questions will be: But why bother at all? Why not just build the
piece you need *inside* asyncio? Why does this need anything more? The
answer to those questions are wants and needs. If I'm a user that needs
interactive subprocess handling, I want to be able to do something like the
code snippet above. The last thing I need is to have to rewrite the way my
application/script/whatever handles *everything* just because a new
asynchronous IO library has been included in the Python standard library -
it's a bit like selling you a $300 bicycle when you need a $20 wheel for
your scooter.
You don't have to rewrite your whole application. If you only want to
use asyncio event loop in a single function, you can use
loop.run_until_complete(do_login) which blocks until the function
completes. The function is an asynchronous coroutine in fact.
Full example of asynchronous communication with a subprocess (the
python interactive interpreter) using asyncio high-level API:
---
import asyncio.subprocess
import time
import sys
@asyncio.coroutine
def eval_python_async(command, encoding='ascii', loop=None):
proc = yield from asyncio.subprocess.create_subprocess_exec(
sys.executable, -u, -i,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
loop=loop)
# wait for the prompt
buffer = bytearray()
while True:
data = yield from proc.stdout.read(100)
buffer.extend(data)
[Python-Dev] Jython site now seems to redirect to docs.python.org???
I'm not sure if this is a result of the recent website reorg, but www.jython.org seems to be redirecting to docs.python.org for me. Presumably this is an error - where do I report it and/or is it a known issue? Thanks, Paul. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 28 March 2014 20:20, Victor Stinner victor.stin...@gmail.com wrote:
2014-03-28 2:16 GMT+01:00 Josiah Carlson josiah.carl...@gmail.com:
def do_login(...):
proc = subprocess.Popen(...)
current = proc.recv(timeout=5)
last_line = current.rstrip().rpartition('\n')[-1]
if last_line.endswith('login:'):
proc.send(username)
if proc.readline(timeout=5).rstrip().endswith('password:'):
proc.send(password)
if 'welcome' in proc.recv(timeout=5).lower():
return proc
proc.kill()
I don't understand this example. How is it asynchronous? It looks
like blocking calls. In my definition, asynchronous means that you can
call this function twice on two processes, and they will run in
parallel.
Without reading all the reference from PEP 3145 again, I now seem to
recall the problem it was aimed at was the current deadlock warnings
in the subprocess docs - if you're not careful to make sure you keep
reading from the stdout and stderr pipes while writing to stdin, you
can fill up the kernel buffers and deadlock while communicating with
the subprocess. So the asynchronous part is to be able to happily
write large amounts of data to a stdin pipe without fear of deadlock
with a subprocess that has just written large amounts of data to the
stdout or stderr pipes.
So, from the perspective of the user, it behaves like a synchronous
blocking operation, but on the backend it needs to use asynchronous
read and write operations to avoid deadlock. I suspect it would likely
be a relatively thin wrapper around run_until_complete().
Also, as far as where such functionality should live in the standard
library could go, it's entirely possible for it to live in its natural
home of subprocess. To make that work, the core subprocess.Popen
functionality would need to be moved to a _subprocess module, and then
both subprocess and asyncio would depend on that, allowing subprocess
to also depend on asyncio without creating a circular import.
So I'll go back on my original comment - assuming I've now remembered
its intended effects correctly PEP 3145 remains a valid proposal,
independent of (but potentially relying on) asyncio, as the problem it
is designed to solve is all those notes like Do not use stdout=PIPE
or stderr=PIPE with this function. As the pipes are not being read in
the current process, the child process may block if it generates
enough output to a pipe to fill up the OS pipe buffer. in the current
subprocess module by using an asynchronous backend while still
presenting a synchronous API.
And rather than adding a new API, I'd hope it could propose just
getting rid of those warnings by reimplementing the current deadlock
prone APIs on top of run_until_complete() and exploring the
potential consequences for backwards compatibility.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
I'm not sure where the responsibilities of the redesign team end and those of the infrastructure team start, but since the switch to the new site I've been adding anything related to the website parts of python.org to https://github.com/python/pythondotorg/issues If that's not the right place, then I'd also like to know the answer to Paul's where do I report it? question :) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
On 28 March 2014 10:53, Nick Coghlan ncogh...@gmail.com wrote: I'm not sure where the responsibilities of the redesign team end and those of the infrastructure team start, but since the switch to the new site I've been adding anything related to the website parts of python.org to https://github.com/python/pythondotorg/issues Thanks, Nick - I've just raised an issue on there. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Negative timedelta strings
There's a furious discussion going on at the python-list mailing list, about negative timedelta strings: py str(timedelta(0, -1)) '-1 day, 23:59:59' This is documented. It's even documented as being somewhat unusual. I found a tracker item for it, back in July 2010: http://bugs.python.org/issue9430 and that refers back to a discussion on #python-dev, but (as far as I know) that's not archived anywhere. I had a look on the python-dev mailing list around that time, and couldn't find anything. Does anyone remember the rationale for this behaviour? Is it open to debate or is it now cast in stone? (Personally, I'm not looking to reopen the debate, and if I did, it would be on python-ideas.) Any links to the earlier discussion would be helpful. Thanks, -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On the necessity of PEPs [was collections.sortedtree]
On Thu, 27 Mar 2014 20:32:02 +1000 Nick Coghlan ncogh...@gmail.com wrote: Most of the time when I hear people say the PEP process is too difficult, I eventually find that what they really mean is learning the kinds of things that python-dev are likely to be worried about, and ensuring that the PEP adequately addresses their concerns, and listening to feedback, and reconsidering what I actually want, and revising my proposal, such that they eventually say yes is too time consuming. Well, the PEP process *is* difficult and not only because you have to learn the kinds of things that python-dev are likely to be worried about. Getting a PEP accepted for a feature is much more work than getting a feature accepted in the bug tracker. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On the necessity of PEPs [was collections.sortedtree]
On 28 March 2014 21:12, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 27 Mar 2014 20:32:02 +1000 Nick Coghlan ncogh...@gmail.com wrote: Most of the time when I hear people say the PEP process is too difficult, I eventually find that what they really mean is learning the kinds of things that python-dev are likely to be worried about, and ensuring that the PEP adequately addresses their concerns, and listening to feedback, and reconsidering what I actually want, and revising my proposal, such that they eventually say yes is too time consuming. Well, the PEP process *is* difficult and not only because you have to learn the kinds of things that python-dev are likely to be worried about. Getting a PEP accepted for a feature is much more work than getting a feature accepted in the bug tracker. Oh, agreed. It's only the too qualifier that I question - I'm not sure how much easier we could make it before it ceased to serve its filtering purpose. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
On Mar 28, 2014, at 6:57 AM, Paul Moore p.f.mo...@gmail.com wrote: On 28 March 2014 10:53, Nick Coghlan ncogh...@gmail.com wrote: I'm not sure where the responsibilities of the redesign team end and those of the infrastructure team start, but since the switch to the new site I've been adding anything related to the website parts of python.org to https://github.com/python/pythondotorg/issues Thanks, Nick - I've just raised an issue on there. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Probably infrastructure-st...@python.org - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
On 28 March 2014 11:24, Donald Stufft don...@stufft.io wrote: Probably infrastructure-st...@python.org OK, I've emailed them as well. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
-Original Message-
From: Python-Dev [mailto:python-dev-
bounces+kristjan=ccpgames@python.org] On Behalf Of Antoine Pitrou
Sent: 27. mars 2014 15:53
To: python-dev@python.org
Subject: Re: [Python-Dev] collections.sortedtree
On Thu, 27 Mar 2014 08:50:01 -0700
Daniel Stutzbach stutzb...@google.com wrote:
To provide efficient cancellation and removal, a heap implementation
needs some way to efficiently answer What is the current index of this
item?.
There are a couple of ways to achieve that, but they all require more
storage than heapq's list-based approach.
You are right. I was assuming the index is already known.
Regards
Antoine.
Yes. But for rare occurrances, it is annoying that heap.remove(item) is more
expensive than it
needs to be. It is a reasonable operation, just like list.remove() is.
I'll be willing to experiment with extending the heapq. methods to take an
optional map argument.
'map' would be a dict, mapping objects in the heap to indices. If provided,
each of the heapq methouds would
take care to update the map of any objects that it touches with the current
index of the object.
Usage could be something like:
heap = []
map = {}
def insert(i):
heapq.heappush(heap, I, map)
def pop(i):
return heapq.heappop(heap, map)
def remove(i):
heapq.heapdel(heap, map[i], map)
K
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/27/2014 09:16 PM, Josiah Carlson wrote: But here's the thing: I can build enough using asyncio in 30-40 lines of Python to offer something like the above API. The problem is that it really has no natural home. It uses asyncio, so makes no sense to put in subprocess. It doesn't fit the typical asyncio behavior, so doesn't make sense to put in asyncio. The required functionality isn't big enough to warrant a submodule anywhere. Heck, it's even way too small to toss into an external PyPI module. Seems perfect for the Cheesehop to me. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1iucACgkQ+gerLs4ltQ4RygCfQOjBD7jTZU5ILub/sKxGYqH8 8v8AoKkv2ePkRn3X43CpGBQNeB9uNufQ =xgSe -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] libpython added to ABI tracker
Hi, The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. -- Andrey Ponomarenko, NTC IT ROSA. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/27/2014 04:11 AM, Stephen J. Turnbull wrote: Maybe. That depends on if you care about the convenience of folks who have to get new modules past Corporate Security, but it's easier to get an upgrade of the whole shebang. I don't think it's ever really been resolved whether they're a typical case that won't go away or a special group whose special needs should be considered. ISTM that such concerns should be addressed via some kind of paid support contract (a la RHEL), and not used to drive decisions for the underlying FLOSS project. The existence of aggregated resources from such a support organization would then be relevant to the include / exclude discussion: presumably the support organization could fund the maintenance of the otherwise-excluded module based on its customers' paid-for needs. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1jtQACgkQ+gerLs4ltQ5OOgCdHeOjBjpfJ1w5mkAWZsajflWK U3wAmgIxnc7BFIaoouQ0kdkCgoF+lMr3 =yhYm -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libpython added to ABI tracker
Hi, 2014-03-28 9:31 GMT+01:00 Andrey Ponomarenko aponomare...@rosalab.ru: The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. Nice! By the way, would it be possible to add a second page for the stable ABI of Python? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
On Fri, Mar 28, 2014, at 3:28, Paul Moore wrote: I'm not sure if this is a result of the recent website reorg, but www.jython.org seems to be redirecting to docs.python.org for me. Presumably this is an error - where do I report it and/or is it a known issue? This was my fault, though maybe apache 2.2's weird virtual host selection rules can share some of the blame. Fixed now. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
On Fri, Mar 28, 2014 at 7:03 AM, Steven D'Aprano st...@pearwood.info wrote: py str(timedelta(0, -1)) '-1 day, 23:59:59' .. Does anyone remember the rationale for this behaviour? I don't recall any better rationale than what I wrote in the docs: String representations of timedelta objects are normalized similarly to their internal representation. Is it open to debate or is it now cast in stone? I think the barrier for changing str() is lower than that for changing repr(), but I would be against any changes in this area. (I may have had a different view if ISO 8601 syntax for timedeltas was not so ugly. :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On 28 March 2014 16:22, Tres Seaver tsea...@palladion.com wrote: On 03/28/2014 12:18 PM, Tres Seaver wrote: I'm mostly arguing the FLOSS project should feel free to ignore high-maintenance-cost commercial concerns until those concerns bring either blook (funded developer time) or treasure (pooled to pay for the same time) to the table to pay for them. Dangit, s/blook/blood/ Rats. I thought I'd just learned a new word. Regardless of reality, I think blook should be a real word :-) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
On Fri, Mar 28, 2014 at 12:19 PM, Skip Montanaro s...@pobox.com wrote: (*) As an aside (that is, this belongs in a separate thread if you want to discuss it), in my opinion, attempting to support ISO 8601 formatting is pointless without the presence of an anchor datetime. I meant ISO 8601 syntax for durations [1]. For example, P3Y6M4DT12H30M5S represents a duration of three years, six months, four days, twelve hours, thirty minutes, and five seconds. And if you have to ask - no I don't want to see this anywhere near Python stdlib. :-) [1] http://en.wikipedia.org/wiki/ISO_8601#Durations ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
On Fri, 28 Mar 2014 11:19:52 -0500 Skip Montanaro s...@pobox.com wrote: On Fri, Mar 28, 2014 at 11:07 AM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: Is it open to debate or is it now cast in stone? I think the barrier for changing str() is lower than that for changing repr(), but I would be against any changes in this area. (I may have had a different view if ISO 8601 syntax for timedeltas was not so ugly. :-) I think str() should be left alone. It's clear there is no one best str representation for timedelta objects. But at least we could have one that isn't terribly confusing :-) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ISO 8601 durations and datetime.timedelta
Andrew wrote: I meant ISO 8601 syntax for durations [1]. That's exactly what I was referring to. Consider this session: now = datetime.datetime.now() now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) then = now - datetime.timedelta(days=57, hours=12, minutes=12, seconds=12) now datetime.datetime(2014, 3, 28, 12, 4, 38, 517110) then datetime.datetime(2014, 1, 29, 23, 52, 26, 517110) so, the difference is: now - then datetime.timedelta(57, 43932) Given that the timedelta has more than a month's worth of days, how would you describe it using the ISO8601 duration notation without referencing a specific point in time? Conversely, given your example, P3Y6M4DT12H30M5S, how would you convert that into a timedelta without knowing which exact years and months this duration refers to? Timedelta objects are very precise beasts, which is almost certainly why they don't support years and months args in their constructors. This is why I said this deserved a separate topic. Probably on python-ideas. Skip ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2014-03-21 - 2014-03-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4534 (+23) closed 28324 (+51) total 32858 (+74) Open issues with patches: 2090 Issues opened (61) == #15968: Incorporate Tcl/Tk/Tix into the Windows build process http://bugs.python.org/issue15968 reopened by haypo #16047: Tools/freeze no longer works in Python 3 http://bugs.python.org/issue16047 reopened by lemburg #21013: server-specific SSL context configuration http://bugs.python.org/issue21013 opened by pitrou #21014: `1` = `True`; for tutorial docs http://bugs.python.org/issue21014 opened by SamuelMarks #21016: trace: $prefix and $exec_prefix improperly replaced on Fedora http://bugs.python.org/issue21016 opened by mathstuf #21018: [patch] added missing documentation about escaping characters http://bugs.python.org/issue21018 opened by Arun.Persaud #21019: PyMethodDef ml_name is char* instead of const char* http://bugs.python.org/issue21019 opened by h.venev #21020: PyMethodDef ml_doc is char* instead of const char* http://bugs.python.org/issue21020 opened by h.venev #21021: PyMemberDef name is char* instead of const char* http://bugs.python.org/issue21021 opened by h.venev #21022: PyMemberDef doc is char* instead of const char* http://bugs.python.org/issue21022 opened by h.venev #21023: PyTypeObject tp_name is char* instead of const char* http://bugs.python.org/issue21023 opened by h.venev #21024: PyTypeObject tp_doc is char* instead of const char* http://bugs.python.org/issue21024 opened by h.venev #21025: if check_hostname is true, context can't be used for server pu http://bugs.python.org/issue21025 opened by pitrou #21026: Document sitecustomize.py problems with pythonw http://bugs.python.org/issue21026 opened by terry.reedy #21027: difflib new cli interface http://bugs.python.org/issue21027 opened by Claudiu.Popa #21028: ElementTree objects should support all the same methods as Ele http://bugs.python.org/issue21028 opened by rhettinger #21029: IDLE mis-coloring print http://bugs.python.org/issue21029 opened by rhettinger #21030: pip usable only by administrators on Windows http://bugs.python.org/issue21030 opened by Christian.Ullrich #21031: [patch] Add AlpineLinux to the platform module's supported dis http://bugs.python.org/issue21031 opened by Elizacat #21032: Socket leak if HTTPConnection.getresponse() fails http://bugs.python.org/issue21032 opened by vadmium #21033: previous trace function still invoked after sys.settrace() http://bugs.python.org/issue21033 opened by xdegaye #21034: Python docs reference the Distribute package which has been de http://bugs.python.org/issue21034 opened by Jurko.GospodnetiÄ #21035: Python's HTTP server implementations hangs after 16.343 reques http://bugs.python.org/issue21035 opened by chobeiry #21037: add an AddressSanitizer build option http://bugs.python.org/issue21037 opened by neologix #21040: socketserver: use selectors module http://bugs.python.org/issue21040 opened by neologix #21041: pathlib.PurePath.parents rejects negative indexes http://bugs.python.org/issue21041 opened by akira #21042: ctypes.util.find_library() should return full pathname instead http://bugs.python.org/issue21042 opened by Hernan.Grecco #21044: tarfile does not handle file .name being an int http://bugs.python.org/issue21044 opened by seirl #21046: Document formulas used in statistics http://bugs.python.org/issue21046 opened by zach.ware #21047: html.parser.HTMLParser: convert_charrefs should become True by http://bugs.python.org/issue21047 opened by Arfrever #21048: Index 'as' in import and with statements http://bugs.python.org/issue21048 opened by terry.reedy #21052: Consider dropping ImportWarning for empty sys.path_hooks and s http://bugs.python.org/issue21052 opened by brett.cannon #21054: Improve indexing of syntax symbols http://bugs.python.org/issue21054 opened by terry.reedy #21055: Index (augmented) assignment symbols http://bugs.python.org/issue21055 opened by terry.reedy #21056: csv documentation is incorrect http://bugs.python.org/issue21056 opened by NRGunby #21057: TextIOWrapper does not support reading bytearrays or memoryvie http://bugs.python.org/issue21057 opened by nikratio #21059: idle_test.test_warning failure http://bugs.python.org/issue21059 opened by neologix #21060: Better error message for setup.py upload command without sdist http://bugs.python.org/issue21060 opened by eric.araujo #21061: Is contextlib.redirect_stdout reentrant or not? http://bugs.python.org/issue21061 opened by perey #21062: Evalute all import-related modules for best practices http://bugs.python.org/issue21062 opened by brett.cannon #21063: Touch up one-line descriptions of modules for module index http://bugs.python.org/issue21063 opened by
Re: [Python-Dev] Negative timedelta strings
On Fri, Mar 28, 2014 at 11:07 AM, Alexander Belopolsky alexander.belopol...@gmail.com wrote: Is it open to debate or is it now cast in stone? I think the barrier for changing str() is lower than that for changing repr(), but I would be against any changes in this area. (I may have had a different view if ISO 8601 syntax for timedeltas was not so ugly. :-) I think str() should be left alone. It's clear there is no one best str representation for timedelta objects. I think it might be worthwhile to add a method which allows the programmer to format timedelta objects in whatever way s/he sees fit. That would support the ISO 8601 syntax (*), and anything else the programmer things is better than the status quo. Skip (*) As an aside (that is, this belongs in a separate thread if you want to discuss it), in my opinion, attempting to support ISO 8601 formatting is pointless without the presence of an anchor datetime. Otherwise how would you know how far back five months or seven years was? If that's the case, then you might as well add the timedelta to your anchor datetime and just use datetime.strftime(). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/28/2014 11:57 AM, Stephen J. Turnbull wrote: So, let me get this straight: you think that one user should pay Red Hat to vet the package for RHEL, and another user should pay to get it into Ubuntu, and another user to get it into SuSE? And then the distros should all pay lawyers to write contracts to make sure that nobody is paying too much for support in the stdlib when they eventually get it in? (Except the customers, of course, everybody will be happy if *they* pay more than they need to.) No, I'm arguing that *if* the concerns you articulate represent a significant number of corporate* customers (i.e., a market), their interests would be better represented by *some* organization who is paid to reflect them. RHEL and Ubuntu LTS could potentially be contributors to that pool. I'm mostly arguing the FLOSS project should feel free to ignore high-maintenance-cost commercial concerns until those concerns bring either blook (funded developer time) or treasure (pooled to pay for the same time) to the table to pay for them. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1oPAACgkQ+gerLs4ltQ52ZgCg06AobjcZa1lGBDtFzRk6IjEK 6DkAnj33xAkqcDUjLGaT9NP4YtZdkAos =62VL -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ISO 8601 durations and datetime.timedelta
On Fri, Mar 28, 2014 at 1:13 PM, Skip Montanaro s...@pobox.com wrote: Given that the timedelta has more than a month's worth of days, how would you describe it using the ISO8601 duration notation without referencing a specific point in time? Conversely, given your example, P3Y6M4DT12H30M5S, how would you convert that into a timedelta without knowing which exact years and months this duration refers to? Timedelta objects are very precise beasts, which is almost certainly why they don't support years and months args in their constructors. So why would they need to support them in str()? We could make str(timedelta(days=369, hours=12, minutes=30, seconds=5)) return P369DT12H30M5S, but that would hardly be an improvement over print timedelta(days=369, hours=12, minutes=30, seconds=5) 369 days, 12:30:05 and I don't think ISO 8601 allows for negative durations, so it gives us little guidance for the other issue either. My preference would be to leave strict ISO 8601 compliance to 3rd party libraries. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On Fri, 28 Mar 2014 10:45:01 -0400, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/27/2014 09:16 PM, Josiah Carlson wrote: But here's the thing: I can build enough using asyncio in 30-40 lines of Python to offer something like the above API. The problem is that it really has no natural home. It uses asyncio, so makes no sense to put in subprocess. It doesn't fit the typical asyncio behavior, so doesn't make sense to put in asyncio. The required functionality isn't big enough to warrant a submodule anywhere. Heck, it's even way too small to toss into an external PyPI module. Seems perfect for the Cheesehop to me. Indeed. I heard a rumor[*] that there's at least one package in the cheeseshop that consists of a one-liner. On the other hand, we have multiprocessing examples in the docs that are longer than that, so it sounds like a great asyncio example to me, especially given that Victor says we don't have enough examples yet. --David [*] https://pypi.python.org/pypi/first It's not *actually* a one liner, but you could write it as one, and the actual code isn't much heavier :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
*This* is the type of conversation that I wanted to avoid. But I'll answer
your questions because I used to do exactly the same thing.
On Fri, Mar 28, 2014 at 3:20 AM, Victor Stinner victor.stin...@gmail.comwrote:
2014-03-28 2:16 GMT+01:00 Josiah Carlson josiah.carl...@gmail.com:
def do_login(...):
proc = subprocess.Popen(...)
current = proc.recv(timeout=5)
last_line = current.rstrip().rpartition('\n')[-1]
if last_line.endswith('login:'):
proc.send(username)
if proc.readline(timeout=5).rstrip().endswith('password:'):
proc.send(password)
if 'welcome' in proc.recv(timeout=5).lower():
return proc
proc.kill()
I don't understand this example. How is it asynchronous? It looks
like blocking calls. In my definition, asynchronous means that you can
call this function twice on two processes, and they will run in
parallel.
In this context, async means not necessarily blocking. If you didn't
provide a timeout, it would default to 0, which would return immediately
with what was sent and/or received from the subprocess. If you don't
believe me, that's fine, but it prevents meaningful discussion.
Using greenlet/eventlet, you can write code which looks blocking, but
runs asynchronously. But I don't think that you are using greenlet or
eventlet here.
You are right. And you are talking about something that is completely out
of scope.
I take a look at the implementation:
http://code.google.com/p/subprocdev/source/browse/subprocess.py
It doesn't look portable. On Windows, WriteFile() is used. This
function is blocking, or I missed something huge :-) It's much better
if a PEP is portable. Adding time.monotonic() only to Linux would make
the PEP 418 much shorter (4 sentences instead of 10 pages? :-))!
Of course it's not portable. Windows does things differently from other
platforms. That's one of the reasons why early versions required pywin32.
Before you reply to another message, I would encourage you to read the bug,
the pep, and perhaps the recipe I just posted: http://pastebin.com/0LpyQtU5
Or you can try to believe that I have done all of those and believe what I
say, especially when I say that I don't believe that spending a lot of time
worrying about the original patch/recipe and the GSoC entry. They would all
require a lot of work to make reasonably sane, which is why I wrote the
minimal recipe above.
The implementation doesn't look reliable:
def get_conn_maxsize(self, which, maxsize):
# Not 100% certain if I get how this works yet.
if maxsize is None:
maxsize = 1024
...
This constant 1024 looks arbitrary. On UNIX, a write into a pipe may
block with less bytes (512 bytes).
Testing now I seem to be able to send non-reading subprocesses somewhat
arbitrary amounts of data without leading to a block. But I can't test all
Linux installations or verify that I'm correct. But whether or not this
makes sense is moot, as I don't think it should be merged, and I don't
believe anyone thinks it should be merged at this point.
asyncio has a completly different design. On Windows, it uses
overlapped operations with IOCP event loop. Such operation can be
cancelled. Windows cares of the buffering. On UNIX, non-blocking mode
is used with select() (or something faster like epoll) and asyncio
retries to write more data when the pipe (or any file descriptor used
for process stdin/stdoud/stderr) becomes ready (for reading/writing).
asyncio design is more reliable and portable.
More reliable, sure. More portable... only because all of the portability
heavy lifting has been done and included in Python core. That's one other
thing that you aren't understanding - the purpose of trying to have this in
the standard library is so that people can use the functionality (async
subprocesses) on multiple platforms without needing to write it themselves
(poorly), ask on forums of one kind or another, copy and paste from some
recipe posted to the internet, etc. It's a strict increase in the
functionality and usefulness of the Python standard library and has
literally zero backwards compatibility issues.
This is the absolute minimum functionality necessary to make people who
need this functionality happy. No, really. Absolute minimum. Sort of what
asyncore was - the minimum functionality necessary to have async sockets in
Python. Was it dirty? Sure. Was it difficult to use? Some people had
issues. Did it work? It worked well enough that people were making money
building applications based on asyncore (myself included 10 years ago).
I don't see how you can implement asynchronous communication with a
subprocess without the complex machinery of an event loop.
Words can have multiple meanings. The meaning of async in this context is
different from what you believe it to mean, which is part of your
confusion. I tried to address this in my last message, but either you
didn't read that part,
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson josiah.carl...@gmail.comwrote: If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 Seeing this makes *me* feel better. I think it's reasonable to add (some variant) of that to the subprocess module in Python 3.5. It also belongs in the Activestate cookbook. And no, the asyncio module hasn't made it obsolete. Josiah, you sound upset about the whole thing -- to the point of writing unintelligible sentences and passive-aggressive digs at everyone reading this list. I'm sorry that something happened that led you feel that way (if you indeed feel upset or frustrated) but I'm glad that you wrote that code snippet -- it is completely clear what you want and why you want it, and also what should happen next (a few rounds of code review on the tracker). But that PEP? It's just a terrible PEP. It doesn't contain a single line of example code. It doesn't specify the proposed interface, it just describes in way too many sentences how it should work, and contains a whole lot of references to various rants from which the reader is apparently meant to become enlightened. I don't know which of the three authors *really* wrote it, and I don't want to know -- I think the PEP is irrelevant to the proposed feature, which is of put it in the bug tracker and work from there category -- presumably the PEP was written based on the misunderstanding that having a PEP would make acceptance of the patch easier, or because during an earlier bikeshedding round someone said please write a PEP (someone always says that). I propose to scrap the PEP (set the status to Withdrawn) and just work on adding the methods to the subprocess module. If it were me, I'd define three methods, with longer names to clarify what they do, e.g. proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() I.e. add _nonblocking to the method names to clarify that they may return '' when there's nothing available, and have a separate method for reading stderr instead of a flag. And I'd wonder if there should be an unambiguous way to detect EOF or whether the caller should just check for proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable when the other end is closed, and then the write() will fail. But maybe I forget.) But that's all bikeshedding and it can happen on the tracker or directly on the list just as easily; I don't see the need for a PEP. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373
And probably the block should be deindented On Thu, Mar 27, 2014 at 6:48 PM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 27 Mar 2014 17:12:02 +0100 (CET) yury.selivanov python-check...@python.org wrote: +.. classmethod:: Signature.from_callable(obj) + + Return a :class:`Signature` (or its subclass) object for a given callable + ``obj``. This method simplifies subclassing of :class:`Signature`: + + :: + + class MySignature(Signature): + pass + sig = MySignature.from_callable(min) + assert isinstance(sig, MySignature) + This needs a versionadded tag. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
Tres Seaver writes: On 03/27/2014 04:11 AM, Stephen J. Turnbull wrote: Maybe. That depends on if you care about the convenience of folks who have to get new modules past Corporate Security, but it's easier to get an upgrade of the whole shebang. I don't think it's ever really been resolved whether they're a typical case that won't go away or a special group whose special needs should be considered. ISTM that such concerns should be addressed via some kind of paid support contract (a la RHEL), and not used to drive decisions for the underlying FLOSS project. The existence of aggregated resources from such a support organization would then be relevant to the include / exclude discussion: presumably the support organization could fund the maintenance of the otherwise-excluded module based on its customers' paid-for needs. So, let me get this straight: you think that one user should pay Red Hat to vet the package for RHEL, and another user should pay to get it into Ubuntu, and another user to get it into SuSE? And then the distros should all pay lawyers to write contracts to make sure that nobody is paying too much for support in the stdlib when they eventually get it in? (Except the customers, of course, everybody will be happy if *they* pay more than they need to.) Seems to me that putting it into stdlib in the first place makes more sense, if it's going to end up there at all. Another way to put it is, we need a better way to fund support of routine maintenance (ie, the unfun parts) than negotiating it module by module. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/28/2014 12:18 PM, Tres Seaver wrote: I'm mostly arguing the FLOSS project should feel free to ignore high-maintenance-cost commercial concerns until those concerns bring either blook (funded developer time) or treasure (pooled to pay for the same time) to the table to pay for them. Dangit, s/blook/blood/ Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlM1ocMACgkQ+gerLs4ltQ6k8ACgm+ycaOaqZGsefJU5iu9kL4bS 1e4AmQHq/vb4Q6GV/MNuCZQr4HKG/JER =8fLu -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Jython site now seems to redirect to docs.python.org???
On 28 March 2014 13:46, Benjamin Peterson benja...@python.org wrote: This was my fault, though maybe apache 2.2's weird virtual host selection rules can share some of the blame. Fixed now. Thanks Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On Fri, Mar 28, 2014 at 09:20:35AM +, Kristján Valur Jónsson wrote:
I'll be willing to experiment with extending the heapq. methods to take an
optional map argument.
'map' would be a dict, mapping objects in the heap to indices. If provided,
each of the heapq methouds would
take care to update the map of any objects that it touches with the current
index of the object.
Usage could be something like:
heap = []
map = {}
def insert(i):
heapq.heappush(heap, I, map)
def pop(i):
return heapq.heappop(heap, map)
def remove(i):
heapq.heapdel(heap, map[i], map)
If you're going to make heapq more complex, wouldn't it be better to
encapsulate the logic into a Heap class? heapq should remain the same,
and Heap could (if accepted) move into collections. And should this
discussion move to python-ideas?
--
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libpython added to ABI tracker
On Fri, 28 Mar 2014 12:31:59 +0400 Andrey Ponomarenko aponomare...@rosalab.ru wrote: Hi, The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. Thanks. Do note that most of these changes are on private (internal) APIs that normal user code should never invoke / rely on (for example all those APIs starting with an underscore). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum gu...@python.org wrote: On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson josiah.carl...@gmail.comwrote: If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 Seeing this makes *me* feel better. I think it's reasonable to add (some variant) of that to the subprocess module in Python 3.5. It also belongs in the Activestate cookbook. And no, the asyncio module hasn't made it obsolete. Cool. Josiah, you sound upset about the whole thing -- to the point of writing unintelligible sentences and passive-aggressive digs at everyone reading this list. I'm sorry that something happened that led you feel that way (if you indeed feel upset or frustrated) but I'm glad that you wrote that code snippet -- it is completely clear what you want and why you want it, and also what should happen next (a few rounds of code review on the tracker). I'm not always a prat. Something about python-dev brings out parts of me that I thought I had discarded from my personality years ago. Toss in a bit of needing to re-explain ideas that I've been trying to explain for almost 9 years? Frustration + formerly discarded personality traits = uck. That's pretty much why I won't be rejoining the party here regularly, you are all better off without me commenting on 95% of threads like I used to. Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I was when I spend time on this list. That's *my* issue, not yours. That I spent any time redirecting my frustration towards you is BS, and if I could take back the email I sent just before getting Guido's, I would. I would advise everyone to write it off as the ramblings of a surprisingly young, angry old man. Or call me an a-hole. Both are pretty accurate. :) But that PEP? It's just a terrible PEP. It doesn't contain a single line of example code. It doesn't specify the proposed interface, it just describes in way too many sentences how it should work, and contains a whole lot of references to various rants from which the reader is apparently meant to become enlightened. I don't know which of the three authors *really* wrote it, and I don't want to know -- I think the PEP is irrelevant to the proposed feature, which is of put it in the bug tracker and work from there category -- presumably the PEP was written based on the misunderstanding that having a PEP would make acceptance of the patch easier, or because during an earlier bikeshedding round someone said please write a PEP (someone always says that). I propose to scrap the PEP (set the status to Withdrawn) and just work on adding the methods to the subprocess module. I'm not going to argue. The first I read it was 2-3 days ago. If it were me, I'd define three methods, with longer names to clarify what they do, e.g. proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() Easily doable. I.e. add _nonblocking to the method names to clarify that they may return '' when there's nothing available, and have a separate method for reading stderr instead of a flag. And I'd wonder if there should be an unambiguous way to detect EOF or whether the caller should just check for proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable when the other end is closed, and then the write() will fail. But maybe I forget.) But that's all bikeshedding and it can happen on the tracker or directly on the list just as easily; I don't see the need for a PEP. Sounds good. - Josiah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On 2014-03-28 16:39, Paul Moore wrote: On 28 March 2014 16:22, Tres Seaver tsea...@palladion.com wrote: On 03/28/2014 12:18 PM, Tres Seaver wrote: I'm mostly arguing the FLOSS project should feel free to ignore high-maintenance-cost commercial concerns until those concerns bring either blook (funded developer time) or treasure (pooled to pay for the same time) to the table to pay for them. Dangit, s/blook/blood/ Rats. I thought I'd just learned a new word. Regardless of reality, I think blook should be a real word :-) Yes, and it should mean funded developer time on or for an open source project. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement
Am 27.03.2014 22:21, schrieb Ethan Furman: On 03/27/2014 01:44 PM, Guido van Rossum wrote: Accepted. Yay! +1 for that Yay :) Georg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 3/28/2014 12:45 PM, Josiah Carlson wrote: If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 Thank you. The docs gave me the impression that I could simply write proc.stdin and read proc.stdout. I failed with even a simple echo server (on Windows) and your code suggests why. So it does not get lost, I attached your code to http://bugs.python.org/issue18823 My interest is with Idle. It originally ran user code in the same process as the Shell and Editor code. Then Guido added an option to os.spawn a separate process and communicate through a socket connection and the option became the default with same process (requested by -N on the command line) as a backup option. 3.2 switched to using subprocess, but still with a socket. The problem is that the socket connection intermittently fails. Firewalls are, or at least used to be one possible cause, but there are others -- unknown. (While it works, the suggestion to restart with -N is a mystery to people who have never seen a command line.) This is one of the biggest sources of complaints about Idle. A pipe connection method that always worked on Windows, *x, and Mac would be great in itself and would also allow code simplification by removing the -n option. (Roger Serwy has suggested the latter as having two modes makes patching trickier.) The current socket connection must be non-blocking. Even though the exec loop part of the Shell window waits for a response after sending a user statement, everything else is responsive. One can select text in the window, use the menus, or switch to another window. So Idle definitely needs non-blocking write and read. In my ignorance, I have no idea whether the approach in your code or that in Viktor's code is better. Either way, I will appreciate any help you give, whether by writing, reviewing, or testing, to make communication with subprocesses easier and more dependable. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On Fri, Mar 28, 2014 at 12:42 PM, Terry Reedy tjre...@udel.edu wrote: On 3/28/2014 12:45 PM, Josiah Carlson wrote: If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 Thank you. The docs gave me the impression that I could simply write proc.stdin and read proc.stdout. I failed with even a simple echo server (on Windows) and your code suggests why. So it does not get lost, I attached your code to http://bugs.python.org/issue18823 My interest is with Idle. It originally ran user code in the same process as the Shell and Editor code. Then Guido added an option to os.spawn a separate process and communicate through a socket connection and the option became the default with same process (requested by -N on the command line) as a backup option. 3.2 switched to using subprocess, but still with a socket. The problem is that the socket connection intermittently fails. Firewalls are, or at least used to be one possible cause, but there are others -- unknown. (While it works, the suggestion to restart with -N is a mystery to people who have never seen a command line.) This is one of the biggest sources of complaints about Idle. A pipe connection method that always worked on Windows, *x, and Mac would be great in itself and would also allow code simplification by removing the -n option. (Roger Serwy has suggested the latter as having two modes makes patching trickier.) The current socket connection must be non-blocking. Even though the exec loop part of the Shell window waits for a response after sending a user statement, everything else is responsive. One can select text in the window, use the menus, or switch to another window. So Idle definitely needs non-blocking write and read. In my ignorance, I have no idea whether the approach in your code or that in Viktor's code is better. Either way, I will appreciate any help you give, whether by writing, reviewing, or testing, to make communication with subprocesses easier and more dependable. One of my other use-cases for this was using this in *my* editor (PyPE), which I wrote (in 2003) because I lost work in Idle. This lost work was due to the same-process interpreter crashing during an interactive session. IIRC, this is partly what pushed Guido to have Idle use os.spawn() + socket. I ended up using wxPython's built-in external process support at the time, but that's obviously not useful in core Python with Idle :P This is all coming back full circle. :) - Josiah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 3/28/2014 11:35 AM, Josiah Carlson wrote: If it were me, I'd define three methods, with longer names to clarify what they do, e.g. proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() Easily doable. I'd appreciate being notified if you do update/test as described. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 3/28/2014 6:20 AM, Victor Stinner wrote:
Full example of asynchronous communication with a subprocess (the
python interactive interpreter) using asyncio high-level API:
Thank you for writing this. As I explained in response to Josiah, Idle
communicates with a python interpreter subprocess through a socket.
Since making the connection is not dependable, I would like to replace
the socket with the pipes. http://bugs.python.org/issue18823
However, the code below creates a subprocess for one command and one
response, which can apparently be done now with subprocess.communicate.
What I and others need is a continuing (non-blocking) conversion with 1
and only 1 subprocess (see my response to Josiah), and that is much more
difficult. So this code does not do what he claims his will do.
However it is done, I agree with the intent of the PEP to make it much
easier to talk with a subprocess. Victor, if you can rewrite the below
with a run_forever loop that can accept new write-read task pairs and
also make each line read immediately accessible, that would be really
helpful. Post it on the issue above if you prefer.
Another difference from what you wrote below and what Idle does today is
that the shell, defined in idlelib/PyShell.py, does not talk to the
subprocess interpreter directly but to a run supervisor defined in
idlelib/run.py through an rpc protocol ('cmd', 'arg string'). To use the
pipes, the supervisor would grab all input from stdin (instead of the
socket) and exec user code as it does today, or it could be replaced by
a supervisor class with an instance with a name like
_idle_supervisor_3_4_0_ that would be extremely unlikely to clash with
any name created by users.
---
import asyncio.subprocess
import time
import sys
@asyncio.coroutine
def eval_python_async(command, encoding='ascii', loop=None):
proc = yield from asyncio.subprocess.create_subprocess_exec(
sys.executable, -u, -i,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
loop=loop)
# wait for the prompt
buffer = bytearray()
while True:
data = yield from proc.stdout.read(100)
buffer.extend(data)
if buffer.endswith(b' '):
break
proc.stdin.write(command.encode(encoding) + b\n)
yield from proc.stdin.drain()
proc.stdin.close()
output = yield from proc.stdout.read()
output = output.decode(encoding)
output = output.rstrip()
if output.endswith(''):
output = output[:-3].rstrip()
return output
def eval_python(command, timeout=None):
loop = asyncio.get_event_loop()
task = asyncio.Task(eval_python_async(command, loop=loop), loop=loop)
return loop.run_until_complete(asyncio.wait_for(task, timeout))
def test_sequential(nproc, command):
t0 = time.monotonic()
for index in range(nproc):
eval_python(command)
return time.monotonic() - t0
def test_parallel(nproc, command):
loop = asyncio.get_event_loop()
tasks = [asyncio.Task(eval_python_async(command, loop=loop), loop=loop)
for index in range(nproc)]
t0 = time.monotonic()
loop.run_until_complete(asyncio.wait(tasks))
return time.monotonic() - t0
print(1+1 = %r % eval_python(1+1, timeout=1.0))
slow_code = import math; print(str(math.factorial(2)).count('7'))
dt = test_sequential(10, slow_code)
print(Run 10 tasks in sequence: %.1f sec % dt)
dt2 = test_parallel(10, slow_code)
print(Run 10 tasks in parallel: %.1f sec (speed=%.1f) % (dt2, dt/dt2))
# cleanup asyncio
asyncio.get_event_loop().close()
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
To be clear, the proposal for Idle would be to still use the RPC protocol,
but run it over a pipe instead of a socket, right?
On Fri, Mar 28, 2014 at 1:58 PM, Terry Reedy tjre...@udel.edu wrote:
On 3/28/2014 6:20 AM, Victor Stinner wrote:
Full example of asynchronous communication with a subprocess (the
python interactive interpreter) using asyncio high-level API:
Thank you for writing this. As I explained in response to Josiah, Idle
communicates with a python interpreter subprocess through a socket. Since
making the connection is not dependable, I would like to replace the socket
with the pipes. http://bugs.python.org/issue18823
However, the code below creates a subprocess for one command and one
response, which can apparently be done now with subprocess.communicate.
What I and others need is a continuing (non-blocking) conversion with 1 and
only 1 subprocess (see my response to Josiah), and that is much more
difficult. So this code does not do what he claims his will do.
However it is done, I agree with the intent of the PEP to make it much
easier to talk with a subprocess. Victor, if you can rewrite the below with
a run_forever loop that can accept new write-read task pairs and also make
each line read immediately accessible, that would be really helpful. Post
it on the issue above if you prefer.
Another difference from what you wrote below and what Idle does today is
that the shell, defined in idlelib/PyShell.py, does not talk to the
subprocess interpreter directly but to a run supervisor defined in
idlelib/run.py through an rpc protocol ('cmd', 'arg string'). To use the
pipes, the supervisor would grab all input from stdin (instead of the
socket) and exec user code as it does today, or it could be replaced by a
supervisor class with an instance with a name like _idle_supervisor_3_4_0_
that would be extremely unlikely to clash with any name created by users.
---
import asyncio.subprocess
import time
import sys
@asyncio.coroutine
def eval_python_async(command, encoding='ascii', loop=None):
proc = yield from asyncio.subprocess.create_subprocess_exec(
sys.executable, -u, -i,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.STDOUT,
loop=loop)
# wait for the prompt
buffer = bytearray()
while True:
data = yield from proc.stdout.read(100)
buffer.extend(data)
if buffer.endswith(b' '):
break
proc.stdin.write(command.encode(encoding) + b\n)
yield from proc.stdin.drain()
proc.stdin.close()
output = yield from proc.stdout.read()
output = output.decode(encoding)
output = output.rstrip()
if output.endswith(''):
output = output[:-3].rstrip()
return output
def eval_python(command, timeout=None):
loop = asyncio.get_event_loop()
task = asyncio.Task(eval_python_async(command, loop=loop),
loop=loop)
return loop.run_until_complete(asyncio.wait_for(task, timeout))
def test_sequential(nproc, command):
t0 = time.monotonic()
for index in range(nproc):
eval_python(command)
return time.monotonic() - t0
def test_parallel(nproc, command):
loop = asyncio.get_event_loop()
tasks = [asyncio.Task(eval_python_async(command, loop=loop),
loop=loop)
for index in range(nproc)]
t0 = time.monotonic()
loop.run_until_complete(asyncio.wait(tasks))
return time.monotonic() - t0
print(1+1 = %r % eval_python(1+1, timeout=1.0))
slow_code = import math; print(str(math.factorial(2)).count('7'))
dt = test_sequential(10, slow_code)
print(Run 10 tasks in sequence: %.1f sec % dt)
dt2 = test_parallel(10, slow_code)
print(Run 10 tasks in parallel: %.1f sec (speed=%.1f) % (dt2, dt/dt2))
# cleanup asyncio
asyncio.get_event_loop().close()
--
Terry Jan Reedy
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
guido%40python.org
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On Fri, 28 Mar 2014 16:58:25 -0400 Terry Reedy tjre...@udel.edu wrote: On 3/28/2014 6:20 AM, Victor Stinner wrote: Full example of asynchronous communication with a subprocess (the python interactive interpreter) using asyncio high-level API: Thank you for writing this. As I explained in response to Josiah, Idle communicates with a python interpreter subprocess through a socket. Since making the connection is not dependable, I would like to replace the socket with the pipes. http://bugs.python.org/issue18823 However, the code below creates a subprocess for one command and one response, which can apparently be done now with subprocess.communicate. What I and others need is a continuing (non-blocking) conversion with 1 and only 1 subprocess (see my response to Josiah), and that is much more difficult. So this code does not do what he claims his will do. Why don't you use multiprocessing or concurrent.futures? They have everything you need for continuous conversation between processes. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
Alexander Belopolsky wrote: I meant ISO 8601 syntax for durations [1]. ISO 8601 doesn't seem to define a representation for negative durations, though, so it wouldn't solve the original problem. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
Le 28 mars 2014 21:59, Terry Reedy tjre...@udel.edu a écrit : On 3/28/2014 6:20 AM, Victor Stinner wrote: Full example of asynchronous communication with a subprocess (the python interactive interpreter) using asyncio high-level API: However, the code below creates a subprocess for one command and one response, which can apparently be done now with subprocess.communicate. What I and others need is a continuing (non-blocking) conversion with 1 and only 1 subprocess (see my response to Josiah), and that is much more difficult. So this code does not do what he claims his will do. I tried to write the shortest example showing how to read and send data and how to make the call blocking. It's different to communicate() because write occurs after the first read. It should be quite easy to enhance my example to execute more commands. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On Mar 26, 2014, at 1:31 PM, Marko Rauhamaa ma...@pacujo.net wrote: I have made a full implementation of a balanced tree and would like to know what the process is to have it considered for inclusion in Python 3. To summarize, the implementation closely parallels dict() features and resides in _collectionsmodule.c under the name collections.sortedtree. It uses solely the operator to compare keys. I have chosen the AVL tree as an implementation technique. FWIW, I think there may be room for a sorted collection somewhere in the standard library. As others have said, the best place to start is by putting a module on PyPi to let it mature and to compete with other approaches to the problem. Here are a few random thoughts on the over-all idea: * An AVL balanced tree isn't the only solution or necessarily the best solution to the problem. Tree nodes tend to take more space than denser structures and they have awful cache locality (these are the same reasons that deques use doubly-linked blocks rather than a plain doubly linked lists). * Another approach I have experimented with uses lazy sorting. That lets insertion be an O(1) step and incurs a one-time sorting cost upon the next lookup (and because Timsort exploits partial orderings, this can be very cheap). A lazy sorted list is dense and sequentially ordered in memory (reducing space overhead, taking full advantage of cache locality and memory controller auto-prefetch, and providing fast iteration speed by not needing to chase pointers). The lazy sort approach works best in applications that spend most of the time doing lookups and have only infrequent deletions and insertions. * The name of the tool probably should not be sortedtree. Ideally, the tool should be named for what it does, not how it does it (for the most part, users don't need to know whether the underlying implementation is a red-black tree, b-tree, judy array, sqlite database, or lazy list). That is why (I think) that Python dicts are called dicts rather than hash tables (the implementation) or an associative array (the computer science term for the abstract datatype). * There are plenty of data structures that have had good utility and popularity outside of the standard library. I that it is a good thing that blists, numpy arrays, databases, and panda dataframes all live outside the standard library. Those tools are easier to maintain externally and it is easier for you to keep control over the design. Remember the saying, the standard library is where code goes to die (and I would add that it should already be mature (or nearly dead) by the time it gets there). * That said, it is a reasonable possibility that the standard library would benefit from some kind sorted collection (the idea comes up from time to time). Raymond Hettinger___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
If it makes you feel any better, I spent an hour this morning building a 2-function API for Linux and Windows, both tested, not using ctypes, and not even using any part of asyncio (the Windows bits are in msvcrt and _winapi). It works in Python 3.3+. You can see it here: http://pastebin.com/0LpyQtU5 - Josiah On Fri, Mar 28, 2014 at 1:09 AM, Paul Moore p.f.mo...@gmail.com wrote: On 28 March 2014 05:09, Josiah Carlson josiah.carl...@gmail.com wrote: So yeah. Someone want to make a decision? Tell me to write the docs, I will. Tell me to go take a long walk off a short pier, I'll thank you for your time and leave you alone. I had a need for this a few years ago. It's messy to do on Windows (ctypes callouts to PeekNamedPipe to check if you can read from the process without blocking). So I would like to see a recipe for this, (even if it's likely to be another few years before I ever need it again :-)). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
On Fri, Mar 28, 2014 at 5:19 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: ISO 8601 doesn't seem to define a representation for negative durations, though, so it wouldn't solve the original problem. Aside from the horribleness of the ISO 8601 notation for a duration, it's best not to confuse the notions of duration and delta. Notionally, a delta contains more information than a duration. -Fred -- Fred L. Drake, Jr.fred at fdrake.net A storm broke loose in my mind. --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
Raymond Hettinger raymond.hettin...@gmail.com: * An AVL balanced tree isn't the only solution or necessarily the best solution to the problem. Tree nodes tend to take more space than denser structures and they have awful cache locality (these are the same reasons that deques use doubly-linked blocks rather than a plain doubly linked lists). Maybe. The key is the API. The implementation underneath should be changeable. For example, Jython would probably use SortedTree to implement it. Performance tests should help decide when an implementation is switched for a more efficient one. In some of my tests, I haven't seen any significant performance differences between RB trees and AVL trees, for example. The blist implementation, which I have taken a quick glance at, buys cache locality at the price of block copying; I have no data to decide if the tradeoff is a good one. The main thing, IMO, is to get one sorted dictionary in. * The name of the tool probably should not be sortedtree. Correct. That was a mistake on the Subject line. In the code, it's sorteddict. * That said, it is a reasonable possibility that the standard library would benefit from some kind sorted collection (the idea comes up from time to time). Yes. As a user, I have craved for an implementation, which is readily available in Java and the linux kernel, for example. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Negative timedelta strings
Greg Ewing greg.ew...@canterbury.ac.nz: ISO 8601 doesn't seem to define a representation for negative durations, though, so it wouldn't solve the original problem. XSD uses ISO 8601 durations and allows a sign before the initial P. It would appear PT1M means 60 or 61 seconds. P1D means 23, 24 or 25 hours. P1M means 28..31 days etc. Timedelta would have no option but to stick to seconds: P29389453.2345S but then, why not simply use a number: 29389453.2345 Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen
On 28/03/2014 06:35 pm, Josiah Carlson wrote: If it were me, I'd define three methods, with longer names to clarify what they do, e.g. proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() Easily doable. To implement write_nonblocking() on Windows, do you intend to use SetNamedPipeHandleState() with PIPE_NOWAIT? The documentation discourages using this: Note that nonblocking mode is supported for compatibility with Microsoft LAN Manager version 2.0 and should not be used to achieve asynchronous input and output (I/O) with named pipes. And I guess you will need to use a poll/sleep loop to simulate blocking or multiplexing. If you want expect-like behaviour then you need some sort of multiplexing. -- RIchard** ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3
On 03/27/2014 04:26 AM, Nick Coghlan wrote: On 27 March 2014 20:47, Victor Stinner victor.stin...@gmail.com wrote: The PEP 461 looks good to me. It's a nice addition to Python 3.5 and the PEP is well defined. +1 from me as well. One minor request is that I don't think the rationale for rejecting numbers from %s is complete [...] Changed to - In particular, ``%s`` will not accept numbers nor ``str``. ``str`` is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: - what makes a number is fuzzy (float? Decimal? Fraction? some user type?) - allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs '3.14') - given the nature of wire formats, explicit is definitely better than implicit - Note: I fixed a typo in your PEP (reST syntax). I also committed a couple of markup tweaks, since it seemed easier to just fix them than explain what was broken. Thanks to both of you for that. However, there are also two dead footnotes (4 5), which I have left alone - I'm not sure if the problem is a missing reference, or if the footnote can go away now. Fixed. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
Marko Rauhamaa ma...@pacujo.net: For example, Jython would probably use SortedTree to implement it. That word just keeps coming out of my keyboard. The Java class is of course the TreeMap. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libpython added to ABI tracker
On 29 March 2014 02:25, Antoine Pitrou solip...@pitrou.net wrote: On Fri, 28 Mar 2014 12:31:59 +0400 Andrey Ponomarenko aponomare...@rosalab.ru wrote: Hi, The libpython library has been added to the ABI tracker: http://upstream-tracker.org/versions/python.html The page lists library versions and changes in API/ABI. Thanks. Do note that most of these changes are on private (internal) APIs that normal user code should never invoke / rely on (for example all those APIs starting with an underscore). That's where having the stable ABI tracked as well would really come into its own, as any regressions in that *are* a big deal. I also consider it a very good sign that the maintenance releases are very stable from an ABI perspective - that's exactly what it supposed to happen :) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On 29 March 2014 01:57, Stephen J. Turnbull step...@xemacs.org wrote: Another way to put it is, we need a better way to fund support of routine maintenance (ie, the unfun parts) than negotiating it module by module. Yes, yes we do, and there *are* people working on that (see https://wiki.python.org/moin/PythonSoftwareFoundation/BoardCandidates2014) :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: shutdown concurrent.futures.ProcessPoolExecutor
This is a follow up of: https://groups.google.com/forum/#!topic/python-tulip/91NCCqV4SFs According to the information I collected so far it seems it's not possible (or at least very hard) to cleanly shutdown a process pool and all its workers in case of KeyboardInterrupt / SIGINT. Literally, what I would like to do is this: - intercept SIGINT form the main process and/or from each worker process - send SIGINT to all workers - wait for them to finish with a timeout - finally exit Whereas it exists a solution based on multiprocessing: http://noswap.com/blog/python-multiprocessing-keyboardinterrupt ...concurrent.futures.ProcessPoolExecutor does not expose all the necessary bits to emulate it. By opening this thread I hope either to find a solution and document it or if it turns out that it is simply not possible to do this via concurrent.futures then discuss whether it's the case to expose more ProcessPoolExecutor APIs, because right now the interface it offers is pretty limited compared to multiprocessing.Pool: http://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.pool.Pool In particular, it won't let you pass an initializer function to pass to multiprocessing.Pool nor terminate() the process pool (only wait() for it). -- Forwarded message -- From: Guido van Rossum gu...@python.org Date: Sat, Mar 29, 2014 at 12:05 AM Subject: Re: [python-tulip] How to cleanly shutdown the IO loop when using run_in_executor()? To: Giampaolo Rodola' g.rod...@gmail.com Cc: python-tulip python-tu...@googlegroups.com You're going to have to move the discussion to python-dev or the python issue tracker then. On Fri, Mar 28, 2014 at 4:02 PM, Giampaolo Rodola' g.rod...@gmail.comwrote: On Wed, Mar 26, 2014 at 7:35 PM, Guido van Rossum gu...@python.orgwrote: Another thing to investigate might be how the executor creates the processes, and if anything happens to signals there. After some further digging this seems to be related with the problem at hand. According to this: http://noswap.com/blog/python-multiprocessing-keyboardinterrupt ...it appears a feasible solution is to prevent workers to ever receive KeyboardInterrupt and have the main process shutdown the pool via pool.terminate() and finally pool.join(). Unfortunately concurrent.futures.ProcessPoolExecutor does not expose all the necessary bits to do that (multiprocessing.Pool(initializer=...) argument and terminate() method). I also suspect that in order to emulate the suggested solution the underlying Process instance should be daemonized, similarly to what multiprocessing.Pool does: http://hg.python.org/cpython/file/3567d7ebd382/Lib/multiprocessing/pool.py#l222 I wonder whether concurrent.futures.ProcessPoolExecutor would be better off exposing more APIs in order to facilitate tasks like these. -- Giampaolo - http://grodola.blogspot.com -- --Guido van Rossum (python.org/~guido) -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] collections.sortedtree
On Fri, Mar 28, 2014 at 5:48 PM, Daniel Stutzbach stutzb...@google.com wrote: On Fri, Mar 28, 2014 at 2:54 PM, Marko Rauhamaa ma...@pacujo.net wrote: The blist implementation, which I have taken a quick glance at, buys cache locality at the price of block copying; I have no data to decide if the tradeoff is a good one. There's actually a compile-time parameter so that you can tune the tradeoff by adjusting how wide each of blist's nodes are. Like most tradeoffs, whether it's good or bad depends on what you're trying to do. RedBlack trees are very similar to a B-tree with a node-width of 4: http://en.wikipedia.org/wiki/Red%E2%80%93black_tree#Analogy_to_B-trees_of_order_4 blist's sorteddict is written in Python on top of the blist type (which is written in C). Because of the Python layer, it's slower by a constant factor compared to pure-C implementations in microbenchmarks. It grieves me a bit to say this, and blist and blist's sorteddict are probably a good tool for an appropriate job, but blist's sorteddict type is quite a bit slower than several pure python implementations of dictionaries with ordered keys. In my testing blist.sorteddict was dead last for random keys, and wasn't last but was still significantly underperforming for sequential keys (outperforming only binary tree and scapegoat tree, behind all others): http://stromberg.dnsalias.org/~strombrg/python-tree-and-heap-comparison/2014-03-18/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
