Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

By digging into the internals of a subprocess produced by Popen(), you can
write in a blocking manner to the stdin pipe, and read in a blocking manner
from the stdout/stderr pipe(s). For scripting most command-line operations,
the lack of timeouts and the ability to *stop* trying to read is as
important as being able to spawn an external process. It kind-of kills that
side of the usefulness of Python as a tool for scripting.

The question is not whether or not a user of Python can dig into the
internals, make some calls, then get it to be non-blocking - the existence
of two different patches to do so (the most recent of which is from 4 1/2
years ago) shows that it *can* be done. The question is whether or not the
desire for the functionality warrants having functions or methods to
perform these operations in the standard library.

I and others have claimed that it should go into the standard library.
Heck, there was enough of a push that Eric got paid to write his version of
the functionality for a GSoC project in 2009. There has even been activity
on the bug itself unrelated to deferring discussions as recently as May
2012 (after which activity seems to have paused for reasons I don't know).
Some people have raised reasonable questions about the API and
implementation, but no one is willing to offer an alternative API that they
think would be better, so discussions about implementation of a
non-existent API for inclusion are moot.


But honestly, I have approximately zero faith that what I say or do will
lead to the inclusion of any changes to the subprocess module. Which is why
I'm offering to write a short example that uses asyncio for inclusion in
the docs. It's not what I've wanted for almost 9 years, but at least it has
a chance of actually happening. I'll take a chance at updating the docs
instead of a 3 to 9 month bikeshedding just to lead to rejection any day.


So yeah. Someone want to make a decision? Tell me to write the docs, I
will. Tell me to go take a long walk off a short pier, I'll thank you for
your time and leave you alone.

 - Josiah



On Thu, Mar 27, 2014 at 7:18 PM, Terry Reedy  wrote:

> On 3/27/2014 9:16 PM, Josiah Carlson wrote:
>
>> You don't understand the point because you don't understand the feature
>> request or PEP. That is probably my fault for not communicating the
>> intent better in the past. The feature request and PEP were written to
>> offer something like the below (or at least enough that the below could
>> be built with minimal effort):
>>
>> def do_login(...):
>>  proc = subprocess.Popen(...)
>>  current = proc.recv(timeout=5)
>>  last_line = current.rstrip().rpartition('\n')[-1]
>>  if last_line.endswith('login:'):
>>  proc.send(username)
>>  if proc.readline(timeout=5).rstrip().endswith('password:'):
>>  proc.send(password)
>>  if 'welcome' in proc.recv(timeout=5).lower():
>>  return proc
>>  proc.kill()
>>
>> The API above can be very awkward (as shown :P ), but that's okay. From
>> those building blocks a (minimally) enterprising user would add
>> functionality to suit their needs. The existing subprocess module only
>> offers two methods for *any* amount of communication over pipes with the
>> subprocess: check_output() and communicate(), only the latter of which
>> supports sending data (once, limited by system-level pipe buffer
>> lengths). Neither allow for nontrivial interactions from a single
>> subprocess.Popen() invocation.
>>
>
> According to my reading of the doc, one should (in the absence of
> deadlocks, and without having timeouts) be able to use proc.stdin.write and
> proc.stdout.read. Do those not actually work?
>
>
>
>  The purpose was to be able to communicate
>
>> in a bidirectional manner with a subprocess without blocking, or
>> practically speaking, blocking with a timeout. That's where the "async"
>> term comes from. Again, there was never any intent to have the
>> functionality be part of asyncore or any other asynchronous sockets
>> framework, which is why there are no handle_*() methods, readable(),
>> writable(), etc.
>>
>> Your next questions will be: But why bother at all? Why not just build
>> the piece you need *inside* asyncio? Why does this need anything more?
>> The answer to those questions are wants and needs. If I'm a user that
>> needs interactive subprocess handling, I want to be able to do something
>> like the code snippet above. The last thing I need is to have to rewrite
>> the way my application/script/whatever handles *everything* just because
>> a new asynchronous IO library has been included in the Python standard
>> library - it's a bit like selling you a $300 bicycle when you need a $20
>> wheel for your scooter.
>>
>> That there *now* exists the ability to have async subprocesses as part
>> of asyncio is a fortunate happenstance, as the necessary underlying
>> tools for building the above now exist in the standard library. It's a
>> ma

Re: [Python-Dev] collections.sortedtree

[Chris Angelico]

On Fri, Mar 28, 2014 at 3:26 PM, Stephen J. Turnbull  wrote:
> This is at least some convenience for *all*
> users, and a near necessity for some strictly controlled environments.
> In the latter, the "gatekeeper" is all too likely to say "PyPI?  Bring
> us a CTO signoff that 'this module is essential', or forget it".

Don't forget, of course, that there is a middle ground. Something
that's really REALLY awesome on PyPI but isn't in the stdlib might be
packaged by various Linux distros. If you use Debian, typing
"apt-cache pkgnames python-" (or using tab completion on an "apt-get
install") will show you a veritable ton of packages, many of which are
originally off PyPI. Even if you don't actually use that distribution,
it might be worth telling the gatekeeper that Debian and/or Red Hat
have indicated trust for some particular version of some particular
package, which might help get over that hump. But it's still going to
be a much bigger hump than "it's in CPython 3.4".

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Stephen J. Turnbull]

Maciej Fijalkowski writes:

 > I just find "my company is stupid

I find labeling companies as "stupid", merely because they are
cautious about what external code they allow their developers to
depend on, unacceptable.  And I'll go to the trouble of explaining
why.

 > so let's work around it by putting stuff to python standard
 > library" unacceptable argument for python-dev and all the python
 > community.

Then you're completely missing the point.  There are two issues in
including code in the Python standard library.  The first is technical
excellence.  You need to hurdle a certain bar, or the code won't go
in.  Furthermore, this bar includes a comprehensive set of regression
tests.  This implies that (1) human review of the code is on average
much easier for CPython stdlib code than for PyPI code, and (2) a
claim that no relevant-to-the-company behavior has actually changed is
(again, on average) much more verifiable and plausible for CPython
than for PyPI code.  This is at least some convenience for *all*
users, and a near necessity for some strictly controlled environments.
In the latter, the "gatekeeper" is all too likely to say "PyPI?  Bring
us a CTO signoff that 'this module is essential', or forget it".

The second is a purely economic tradeoff: is the value of Python
aggregated across all our users better enhanced by restricting
additions to the stdlib, and thus reducing future maintenance effort
and (presumably) increasing the rate at which Python code is improved,
or by improving the batteries that are included (with the opposite
effects)?  That's a judgment call.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Stephen J. Turnbull]

Maciej Fijalkowski writes:
 > On Thu, Mar 27, 2014 at 10:11 AM, Stephen J. Turnbull
 >  wrote:

 > > Maybe.  That depends on if you care about the convenience of folks who
 > > have to get new modules past Corporate Security, but it's easier to
 > > get an upgrade of the whole shebang.  I don't think it's ever really
 > > been resolved whether they're a "typical case that won't go away" or a
 > > special group whose special needs should be considered.
 > >
 > > Steve
 > 
 > And random pieces of C included in the standard library can be
 > shuffled under the carpet under the disguise of upgrade or what are
 > you suggesting?

CPython doesn't do stuff like that; C code in CPython gets careful
review, and can hardly be called "random".  PyPI packages do do stuff
like that.  Not all of them by far, but to tell the difference you
need to review line-by-line.

So I'm simply saying (as Nick did) that it is often easier to get
changes past a corporate bureaucracy if they are certified by
inclusion in CPython (not to exclude PyPy or Jython, but the corporate
consideration is about a specific distribution), than if it's a random
package on PyPI.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Stephen J. Turnbull]

Nick Coghlan writes:
 > On 27 March 2014 18:11, Stephen J. Turnbull  wrote:

 > > I don't think it's ever really been resolved whether they're a
 > > "typical case that won't go away" or a special group whose
 > > special needs should be considered.

 > That said, getting approval is definitely easier when the request is
 > to trust "the Python Software Foundation" vs "J. Random Programmer's
 > account on GitHub", so yes, it still counts in favour of sufficiently
 > compelling stdlib additions.

I note that this *still* doesn't resolve the alternatives presented
above.

I hasten to add, it doesn't need to be done *now*.  Yes, I do think it
matters.

Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Terry Reedy]

On 3/27/2014 9:16 PM, Josiah Carlson wrote:

You don't understand the point because you don't understand the feature
request or PEP. That is probably my fault for not communicating the
intent better in the past. The feature request and PEP were written to
offer something like the below (or at least enough that the below could
be built with minimal effort):

def do_login(...):
 proc = subprocess.Popen(...)
 current = proc.recv(timeout=5)
 last_line = current.rstrip().rpartition('\n')[-1]
 if last_line.endswith('login:'):
 proc.send(username)
 if proc.readline(timeout=5).rstrip().endswith('password:'):
 proc.send(password)
 if 'welcome' in proc.recv(timeout=5).lower():
 return proc
 proc.kill()

The API above can be very awkward (as shown :P ), but that's okay. From
those building blocks a (minimally) enterprising user would add
functionality to suit their needs. The existing subprocess module only
offers two methods for *any* amount of communication over pipes with the
subprocess: check_output() and communicate(), only the latter of which
supports sending data (once, limited by system-level pipe buffer
lengths). Neither allow for nontrivial interactions from a single
subprocess.Popen() invocation.


According to my reading of the doc, one should (in the absence of 
deadlocks, and without having timeouts) be able to use proc.stdin.write 
and proc.stdout.read. Do those not actually work?



 The purpose was to be able to communicate

in a bidirectional manner with a subprocess without blocking, or
practically speaking, blocking with a timeout. That's where the "async"
term comes from. Again, there was never any intent to have the
functionality be part of asyncore or any other asynchronous sockets
framework, which is why there are no handle_*() methods, readable(),
writable(), etc.

Your next questions will be: But why bother at all? Why not just build
the piece you need *inside* asyncio? Why does this need anything more?
The answer to those questions are wants and needs. If I'm a user that
needs interactive subprocess handling, I want to be able to do something
like the code snippet above. The last thing I need is to have to rewrite
the way my application/script/whatever handles *everything* just because
a new asynchronous IO library has been included in the Python standard
library - it's a bit like selling you a $300 bicycle when you need a $20
wheel for your scooter.

That there *now* exists the ability to have async subprocesses as part
of asyncio is a fortunate happenstance, as the necessary underlying
tools for building the above now exist in the standard library. It's a
matter of properly embedding the asyncio-related bits inside a handful
of functions to provide something like the above, which is what I was
offering to write. But why not keep working on the subprocess module?
Yep. Tried that. Coming up on 9 years since I created the feature
request and original Activestate recipe. To go that route is going to be
2-3 times as much work as has already been dedicated to get somewhere
remotely acceptable for inclusion in Python 3.5, but more likely,
subsequent rejection for similar reasons why it has been in limbo.

But here's the thing: I can build enough using asyncio in 30-40 lines of
Python to offer something like the above API. The problem is that it
really has no natural home. It uses asyncio, so makes no sense to put in
subprocess. It doesn't fit the typical asyncio behavior, so doesn't make
sense to put in asyncio. The required functionality isn't big enough to
warrant a submodule anywhere. Heck, it's even way too small to toss into
an external PyPI module. But in the docs? It would show an atypical, but
not wholly unreasonable use of asyncio (the existing example already
shows what I would consider to be an atypical use of asyncio). It would
provide a good starting point for someone who just wants/needs something
like the snippet above. It is *yet another* use-case for asyncio. And it
could spawn a larger library for offering a more fleshed-out
subprocess-related API, though that is probably more wishful thinking on
my part than anything.

  - Josiah



On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner
mailto:victor.stin...@gmail.com>> wrote:

2014-03-27 22:52 GMT+01:00 Josiah Carlson mailto:josiah.carl...@gmail.com>>:
 > * Because it is example docs, maybe a multi-week bikeshedding
discussion
 > about API doesn't need to happen (as long as "read line", "read X
bytes",
 > "read what is available", and "write this data" - all with
timeouts - are
 > shown, people can build everything else they want/need)

I don't understand this point. Using asyncio, you can read and write a
single byte or a whole line. Using functions like asyncio.wait_for(),
it's easy to add a timeout on such operation.

Victor







--
Terry Jan Reedy

___
Python-D

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 03:10 PM, Greg Ewing wrote:

R. David Murray wrote:

I've done the 'landmark' thing as well, in the string context; that can be
very useful when doing incremental test driven development.  (Granted, you
could do that with __bytes__;


Can't you do it more easily just by wrapping ascii()
around the argument? That seems sufficient for debugging
purposes to me.


The problem there is ascii() still returns unicode (okay, okay, str), so you 
still have to encode it.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Greg Ewing]

R. David Murray wrote:

I've done the 'landmark' thing as well, in the string context; that can be
very useful when doing incremental test driven development.  (Granted, you
could do that with __bytes__;


Can't you do it more easily just by wrapping ascii()
around the argument? That seems sufficient for debugging
purposes to me.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

You don't understand the point because you don't understand the feature
request or PEP. That is probably my fault for not communicating the intent
better in the past. The feature request and PEP were written to offer
something like the below (or at least enough that the below could be built
with minimal effort):

def do_login(...):
proc = subprocess.Popen(...)
current = proc.recv(timeout=5)
last_line = current.rstrip().rpartition('\n')[-1]
if last_line.endswith('login:'):
proc.send(username)
if proc.readline(timeout=5).rstrip().endswith('password:'):
proc.send(password)
if 'welcome' in proc.recv(timeout=5).lower():
return proc
proc.kill()

The API above can be very awkward (as shown :P ), but that's okay. From
those building blocks a (minimally) enterprising user would add
functionality to suit their needs. The existing subprocess module only
offers two methods for *any* amount of communication over pipes with the
subprocess: check_output() and communicate(), only the latter of which
supports sending data (once, limited by system-level pipe buffer lengths).
Neither allow for nontrivial interactions from a single subprocess.Popen()
invocation. The purpose was to be able to communicate in a bidirectional
manner with a subprocess without blocking, or practically speaking,
blocking with a timeout. That's where the "async" term comes from. Again,
there was never any intent to have the functionality be part of asyncore or
any other asynchronous sockets framework, which is why there are no
handle_*() methods, readable(), writable(), etc.

Your next questions will be: But why bother at all? Why not just build the
piece you need *inside* asyncio? Why does this need anything more? The
answer to those questions are wants and needs. If I'm a user that needs
interactive subprocess handling, I want to be able to do something like the
code snippet above. The last thing I need is to have to rewrite the way my
application/script/whatever handles *everything* just because a new
asynchronous IO library has been included in the Python standard library -
it's a bit like selling you a $300 bicycle when you need a $20 wheel for
your scooter.

That there *now* exists the ability to have async subprocesses as part of
asyncio is a fortunate happenstance, as the necessary underlying tools for
building the above now exist in the standard library. It's a matter of
properly embedding the asyncio-related bits inside a handful of functions
to provide something like the above, which is what I was offering to write.
But why not keep working on the subprocess module? Yep. Tried that. Coming
up on 9 years since I created the feature request and original Activestate
recipe. To go that route is going to be 2-3 times as much work as has
already been dedicated to get somewhere remotely acceptable for inclusion
in Python 3.5, but more likely, subsequent rejection for similar reasons
why it has been in limbo.

But here's the thing: I can build enough using asyncio in 30-40 lines of
Python to offer something like the above API. The problem is that it really
has no natural home. It uses asyncio, so makes no sense to put in
subprocess. It doesn't fit the typical asyncio behavior, so doesn't make
sense to put in asyncio. The required functionality isn't big enough to
warrant a submodule anywhere. Heck, it's even way too small to toss into an
external PyPI module. But in the docs? It would show an atypical, but not
wholly unreasonable use of asyncio (the existing example already shows what
I would consider to be an atypical use of asyncio). It would provide a good
starting point for someone who just wants/needs something like the snippet
above. It is *yet another* use-case for asyncio. And it could spawn a
larger library for offering a more fleshed-out subprocess-related API,
though that is probably more wishful thinking on my part than anything.

 - Josiah



On Thu, Mar 27, 2014 at 4:24 PM, Victor Stinner wrote:

> 2014-03-27 22:52 GMT+01:00 Josiah Carlson :
> > * Because it is example docs, maybe a multi-week bikeshedding discussion
> > about API doesn't need to happen (as long as "read line", "read X bytes",
> > "read what is available", and "write this data" - all with timeouts - are
> > shown, people can build everything else they want/need)
>
> I don't understand this point. Using asyncio, you can read and write a
> single byte or a whole line. Using functions like asyncio.wait_for(),
> it's easy to add a timeout on such operation.
>
> Victor
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Kristján Valur Jónsson]

for our stackless socket framework we have the same issue.
Windows provides an opaque timer system where a timer can be cancelled by 
handle.  But on linux one has to
craft one's own.

One thing with this particular use case is that a heapq is overkill for network 
timer.  . For network timers a granularity of one
second should typically be sufficient.  Thus, one can implement a map of future 
times (in quantisized time, e.g. whole seconds) to sets of timers.
A timer is then keyed by its quantisized due time plus its callback.  
Cancellation can then be O(1).

From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames@python.org] 
On Behalf Of Guido van Rossum
Sent: 26. mars 2014 21:42
To: Marko Rauhamaa
Cc: Python-Dev
Subject: Re: [Python-Dev] collections.sortedtree

I haven't felt it, heapq feels natural to me for this use case. :-)
I'm aware of the issue of frequent cancelled timers, but chose to wait and see 
rather than preemptively fix it (only so many hours in a day). IIRC pyftplib 
has a clever cleanup algorithm that we can easily add if that usage pattern 
becomes popular.

On Wed, Mar 26, 2014 at 2:36 PM, Marko Rauhamaa 
mailto:ma...@pacujo.net>> wrote:
Guido van Rossum mailto:gu...@python.org>>:

> Actually, the first step is publish it on PyPI, the second is to get a
> fair number of happy users there. The bar for getting something
> included into the stdlib is pretty high -- you need to demonstrate
> that there is a need *and* that having it as a 3rd party module is a
> problem.
I hear you about the process.

About the need part, I'm wondering if you haven't felt it in
implementing the timers for asyncio. I have had that need in several
network programming projects and have ended up using my AVL tree
implementation (C and Python).

Well, time will tell if frequent canceled timers end up piling up the
heap queue.




___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Victor Stinner]

2014-03-27 22:52 GMT+01:00 Josiah Carlson :
> * Because it is example docs, maybe a multi-week bikeshedding discussion
> about API doesn't need to happen (as long as "read line", "read X bytes",
> "read what is available", and "write this data" - all with timeouts - are
> shown, people can build everything else they want/need)

I don't understand this point. Using asyncio, you can read and write a
single byte or a whole line. Using functions like asyncio.wait_for(),
it's easy to add a timeout on such operation.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Victor Stinner]

Hi,

2014-03-27 22:52 GMT+01:00 Josiah Carlson :
> ... but I never made an effort to get it practically working
> with asyncore - primarily because such would be functionally impossible on
> Windows without a lot of work to pull in a chunk of what was pywin32
> libraries (at the time, Windows was a primary target). On the *nix side of
> things, performing the integration would be arguably trivial as select,
> poll, epoll, etc., all deal with pipes the same way as any other file
> handles (on-disk files, domain sockets, network sockets, etc.), with OS X
> being the exception.

You should definitively take a look at asyncio. It handles sockets
*and* pipes on all platforms, and even character devices (PTY) on some
platforms. (The current status is still unclear to me, especially
regarding to the "non blocking" flag of the PTY.) On Windows, asyncio
uses IOCP.

asyncio.subprocess solves also an old issue related to polling:
subprocess.wait(timeout) uses polling because it was not possible to
register an handler for SIGCHLD handler without breaking backward
compatibility. asyncio supports also signals as well.

> As an alternative, easily 95% of what most people would use this for can be
> written as an example using the asyncio module and included in the docs just
> after (or replacing)
> http://docs.python.org/3/library/asyncio-subprocess.html#example . Adding a
> reference to the subprocess module docs to point off to the asyncio
> subprocess example docs would get people a copy/paste snippet that they can
> include and update to their heart's content.

Yeah, a link should be added from the subprocess module to the
asyncio.subprocess module (the module, not the example). FYI the
asyncore doc now has this note:

"Note: This module exists for backwards compatibility only. For new
code we recommend using asyncio."

I opened the following issue for the "ls | wc -l" feature request:
http://bugs.python.org/issue21080

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Antoine Pitrou]

On Thu, 27 Mar 2014 11:57:35 -0700
Ethan Furman  wrote:
> On 03/27/2014 11:41 AM, Guido van Rossum wrote:
> > Much better, but I'm still not happy with including %s at all. Otherwise 
> > it's accept-worthy. (How's that for pressure. :-)
> 
> FWIW, I feel the same, but the need for compatible 2/3 code bases is real.
> 
> Hey, how's this?  We'll let %s in, but immediately deprecate it.  ;)  Of 
> course, we won't remove it until Python IV.

I vote for an environment variable-controlled feature activation
(or with a registry key under Windows) ;)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Antoine Pitrou]

On Thu, 27 Mar 2014 18:47:59 +
Brett Cannon  wrote:
> On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum  wrote:
> 
> > Much better, but I'm still not happy with including %s at all. Otherwise
> > it's accept-worthy. (How's that for pressure. :-)
> >
> 
> But if we only add %b and leave out %s then how is this going to lead to
> Python 2/3 compatible code since %b is not in Python 2? Or am I
> misunderstanding you?

I think we have reached a point where adding porting-related facilities
in 3.5 may actually slow down the pace of porting, rather than
accelerate it (because people will then wait for 3.5 to start porting
stuff).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for subprocess.popen

[Josiah Carlson]

Hopping in to give my take on this, which I've expressed to Antoine
off-list.

When I first built the functionality about 8.5-9 years ago, I personally
just wanted to be able to build something that could replace some of
Expect: http://expect.sourceforge.net/ . The original and perhaps current
API of the GSoC patch were inspired by my experience with asyncore (hence
send() and recv() methods), but I never made an effort to get it
practically working with asyncore - primarily because such would be
functionally impossible on Windows without a lot of work to pull in a chunk
of what was pywin32 libraries (at the time, Windows was a primary target).
On the *nix side of things, performing the integration would be arguably
trivial as select, poll, epoll, etc., all deal with pipes the same way as
any other file handles (on-disk files, domain sockets, network sockets,
etc.), with OS X being the exception. A little work would have been
necessary to handle the two readable file handles and one writable file
handle, but it's not that much different than building a proxy. But I
digress.

At this point I still believe that the functionality is useful from a
scriptable interaction perspective, regardless of platform. I don't believe
that being able to natively support the piping of output from one process
to another is necessary, but a convenient future feature. That said,
discussions about the quality of the existing GSoC patch and its API
basically mean that the existing code to implement async subprocesses
within the subprocess module precludes it from an easy or short acceptance
process. And without substantial efforts from one or more people would doom
the feature request and PEP to rejection.

As an alternative, easily 95% of what most people would use this for can be
written as an example using the asyncio module and included in the docs
just after (or replacing)
http://docs.python.org/3/library/asyncio-subprocess.html#example . Adding a
reference to the subprocess module docs to point off to the asyncio
subprocess example docs would get people a copy/paste snippet that they can
include and update to their heart's content.

Benefits to updating the docs:
* It can happen at any time and doesn't need to wait for a 3.5 release (it
can also happily wait)
* No one likes maintaining code, but everyone loves docs (especially if it
documents likely use-cases)
* Because it is example docs, maybe a multi-week bikeshedding discussion
about API doesn't need to happen (as long as "read line", "read X bytes",
"read what is available", and "write this data" - all with timeouts - are
shown, people can build everything else they want/need)
* An example using asyncio is shorter than the modifications to the
subprocess module
* I would celebrate the closing of a feature request I opened in 2005

Aside from discarding code (Eric's and my own), not supporting Python-side
chained pipes, and potentially angering some purists who *needed* this to
be based on the subprocess module, I'm not sure I can think of any
drawbacks. And arguably 2/3 of those drawbacks are imagined.


Let me know your thoughts. If it gets an "okay", I'll come up with some
example code, update the docs, and post a link to the code review in this
thread.

 - Josiah



On Wed, Mar 26, 2014 at 4:55 AM, Victor Stinner wrote:

> Hi,
>
> For your information, asyncio.subprocess.Process is limited. It's not
> possible yet to connect pipes between two processes. Something like
> "cat | wc -l" where the cat stdin comes from Python.
>
> It's possible to enhance the API to implement that, but the timeframe
> was too short to implement it before Python 3.4.
>
> Victor
>
> 2014-03-25 23:19 GMT+01:00 Antoine Pitrou :
> >
> > Hi,
> >
> > On core-mentorship someone asked about PEP 3145 - Asynchronous I/O for
> > subprocess.popen.  I answered that asyncio now has subprocess support
> > (including non-blocking I/O on the three standard stream pipes), so
> > it's not obvious anything else is needed.
> >
> > Should we change the PEP's status to Rejected or Superseded?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement

[Ethan Furman]

On 03/27/2014 01:44 PM, Guido van Rossum wrote:


Accepted.


Yay!


If you're going to commit another change, may I suggest to add, to the section 
stating that %r is not supported, that %a
is usually a suitable replacement for %r?


Done.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 11:41 AM, Guido van Rossum wrote:

Much better, but I'm still not happy with including %s at all. Otherwise it's 
accept-worthy. (How's that for pressure. :-)


FWIW, I feel the same, but the need for compatible 2/3 code bases is real.

Hey, how's this?  We'll let %s in, but immediately deprecate it.  ;)  Of 
course, we won't remove it until Python IV.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement

[Guido van Rossum]

Accepted. Congrats with marshalling yet another quite contentious
discussion, and putting up with my last-minute block-headedness!

If you're going to commit another change, may I suggest to add, to the
section stating that %r is not supported, that %a is usually a suitable
replacement for %r?


On Thu, Mar 27, 2014 at 1:07 PM, Ethan Furman  wrote:

> Requesting pronouncement on PEP 461.  Full text below.
>
> 
> ===
> PEP: 461
> Title: Adding % formatting to bytes and bytearray
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ethan Furman 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2014-01-13
> Python-Version: 3.5
> Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25,
>   2014-03-27
> Resolution:
>
>
> Abstract
> 
>
> This PEP proposes adding % formatting operations similar to Python 2's
> ``str``
> type to ``bytes`` and ``bytearray`` [1]_ [2]_.
>
>
> Rationale
> =
>
> While interpolation is usually thought of as a string operation, there are
> cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and
> the
> work needed to make up for this missing functionality detracts from the
> overall
> readability of the code.
>
>
> Motivation
> ==
>
> With Python 3 and the split between ``str`` and ``bytes``, one small but
> important area of programming became slightly more difficult, and much more
> painful -- wire format protocols [3]_.
>
> This area of programming is characterized by a mixture of binary data and
> ASCII compatible segments of text (aka ASCII-encoded text).  Bringing back
> a
> restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
> writing new wire format code, and in porting Python 2 wire format code.
>
> Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
> formats, and ``FTP`` and ``HTTP`` communications, among many others.
>
>
> Proposed semantics for ``bytes`` and ``bytearray`` formatting
> =
>
> %-interpolation
> ---
>
> All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``,
> ``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently
> added
> to Python 3) will be supported, and will work as they do for str, including
> the padding, justification and other related modifiers (currently ``#``,
> ``0``,
> ``-``, `` `` (space), and ``+`` (plus any added to Python 3)).  The only
> non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a
> synonym for b).
>
> For the numeric codes, the only difference between ``str`` and ``bytes``
> (or
> ``bytearray``) interpolation is that the results from these codes will be
> ASCII-encoded text, not unicode.  In other words, for any numeric
> formatting
> code `%x`::
>
>b"%x" % val
>
> is equivalent to::
>
>("%x" % val).encode("ascii")
>
> Examples::
>
>>>> b'%4x' % 10
>b'   a'
>
>>>> b'%#4x' % 10
>' 0xa'
>
>>>> b'%04X' % 10
>'000A'
>
> ``%c`` will insert a single byte, either from an ``int`` in range(256), or
> from
> a ``bytes`` argument of length 1, not from a ``str``.
>
> Examples::
>
> >>> b'%c' % 48
> b'0'
>
> >>> b'%c' % b'a'
> b'a'
>
> ``%b`` will insert a series of bytes.  These bytes are collected in one of
> two
> ways:
>
>   - input type supports ``Py_buffer`` [4]_?
> use it to collect the necessary bytes
>
>   - input type is something else?
> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
> ``TypeError``
>
> In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
> rejected
> as the string to bytes conversion requires an encoding, and we are
> refusing to
> guess; numbers are rejected because:
>
>   - what makes a number is fuzzy (float? Decimal? Fraction? some user
> type?)
>
>   - allowing numbers would lead to ambiguity between numbers and textual
> representations of numbers (3.14 vs '3.14')
>
>   - given the nature of wire formats, explicit is definitely better than
> implicit
>
> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
> 2/3 code
> bases easier to maintain.  Python 3 only code should use ``%b``.
>
> Examples::
>
> >>> b'%b' % b'abc'
> b'abc'
>
> >>> b'%b' % 'some string'.encode('utf8')
> b'some string'
>
> >>> b'%b' % 3.14
> Traceback (most recent call last):
> ...
> TypeError: b'%b' does not accept 'float'
>
> >>> b'%b' % 'hello world!'
> Traceback (most recent call last):
> ...
> TypeError: b'%b' does not accept 'str'
>
>
> ``%a`` will give the equivalent of
> ``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated
> value.  Use cases include developing a new protocol and writing landmarks
> into the stream; debugging data going into an existing protocol to see if
> the problem is the protocol itself o

[Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Request for Pronouncement

[Ethan Furman]

Requesting pronouncement on PEP 461.  Full text below.

===
PEP: 461
Title: Adding % formatting to bytes and bytearray
Version: $Revision$
Last-Modified: $Date$
Author: Ethan Furman 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2014-01-13
Python-Version: 3.5
Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25,
  2014-03-27
Resolution:


Abstract


This PEP proposes adding % formatting operations similar to Python 2's ``str``
type to ``bytes`` and ``bytearray`` [1]_ [2]_.


Rationale
=

While interpolation is usually thought of as a string operation, there are
cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
work needed to make up for this missing functionality detracts from the overall
readability of the code.


Motivation
==

With Python 3 and the split between ``str`` and ``bytes``, one small but
important area of programming became slightly more difficult, and much more
painful -- wire format protocols [3]_.

This area of programming is characterized by a mixture of binary data and
ASCII compatible segments of text (aka ASCII-encoded text).  Bringing back a
restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
writing new wire format code, and in porting Python 2 wire format code.

Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
formats, and ``FTP`` and ``HTTP`` communications, among many others.


Proposed semantics for ``bytes`` and ``bytearray`` formatting
=

%-interpolation
---

All the numeric formatting codes (``d``, ``i``, ``o``, ``u``, ``x``, ``X``,
``e``, ``E``, ``f``, ``F``, ``g``, ``G``, and any that are subsequently added
to Python 3) will be supported, and will work as they do for str, including
the padding, justification and other related modifiers (currently ``#``, ``0``,
``-``, `` `` (space), and ``+`` (plus any added to Python 3)).  The only
non-numeric codes allowed are ``c``, ``b``, ``a``, and ``s`` (which is a
synonym for b).

For the numeric codes, the only difference between ``str`` and ``bytes`` (or
``bytearray``) interpolation is that the results from these codes will be
ASCII-encoded text, not unicode.  In other words, for any numeric formatting
code `%x`::

   b"%x" % val

is equivalent to::

   ("%x" % val).encode("ascii")

Examples::

   >>> b'%4x' % 10
   b'   a'

   >>> b'%#4x' % 10
   ' 0xa'

   >>> b'%04X' % 10
   '000A'

``%c`` will insert a single byte, either from an ``int`` in range(256), or from
a ``bytes`` argument of length 1, not from a ``str``.

Examples::

>>> b'%c' % 48
b'0'

>>> b'%c' % b'a'
b'a'

``%b`` will insert a series of bytes.  These bytes are collected in one of two
ways:

  - input type supports ``Py_buffer`` [4]_?
use it to collect the necessary bytes

  - input type is something else?
use its ``__bytes__`` method [5]_ ; if there isn't one, raise a 
``TypeError``

In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is rejected
as the string to bytes conversion requires an encoding, and we are refusing to
guess; numbers are rejected because:

  - what makes a number is fuzzy (float? Decimal? Fraction? some user type?)

  - allowing numbers would lead to ambiguity between numbers and textual
representations of numbers (3.14 vs '3.14')

  - given the nature of wire formats, explicit is definitely better than 
implicit

``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 
code
bases easier to maintain.  Python 3 only code should use ``%b``.

Examples::

>>> b'%b' % b'abc'
b'abc'

>>> b'%b' % 'some string'.encode('utf8')
b'some string'

>>> b'%b' % 3.14
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'float'

>>> b'%b' % 'hello world!'
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'str'


``%a`` will give the equivalent of
``repr(some_obj).encode('ascii', 'backslashreplace')`` on the interpolated
value.  Use cases include developing a new protocol and writing landmarks
into the stream; debugging data going into an existing protocol to see if
the problem is the protocol itself or bad data; a fall-back for a serialization
format; or any situation where defining ``__bytes__`` would not be appropriate
but a readable/informative representation is needed [6]_.

Examples::

>>> b'%a' % 3.14
b'3.14'

>>> b'%a' % b'abc'
b"b'abc'"

>>> b'%a' % 'def'
b"'def'"


Unsupported codes
-

``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported.


Compatibility with Python 2
===

As noted above, ``%s`` is being included solely to help ease migration from,
and/or have a single code base with, Python 2.  This is important as there
are

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

I love it!


On Thu, Mar 27, 2014 at 12:11 PM, Ethan Furman  wrote:

> On 03/27/2014 11:59 AM, Guido van Rossum wrote:
>
>>
>> PS. I am not trying to be difficult. I honestly don't understand the use
>> case yet, and the PEP doesn't do much to
>> support it.
>>
>
> How's this?
> 
> 
> Compatibility with Python 2
> ===
>
> As noted above, ``%s`` is being included solely to help ease migration
> from,
> and/or have a single code base with, Python 2.  This is important as there
> are modules both in the wild and behind closed doors that currently use the
> Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s``
> as a bytes interpolator.
>
> However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will
> immediately be deprecated, but not removed until the next major Python
> release.
>
> 
> 
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 11:59 AM, Guido van Rossum wrote:


PS. I am not trying to be difficult. I honestly don't understand the use case 
yet, and the PEP doesn't do much to
support it.


How's this?

Compatibility with Python 2
===

As noted above, ``%s`` is being included solely to help ease migration from,
and/or have a single code base with, Python 2.  This is important as there
are modules both in the wild and behind closed doors that currently use the
Python 2 ``str`` type as a ``bytes`` container, and hence are using ``%s``
as a bytes interpolator.

However, ``%b`` should be used in new, Python 3 only code, so ``%s`` will
immediately be deprecated, but not removed until the next major Python
release.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 11:53 AM, Guido van Rossum wrote:


So what's the use case for Python 2/3 compatible code? IMO the main use case 
for the PEP is simply to be able to
construct bytes from a combination of a template and some input that may 
include further bytes and numbers. E.g. in
asyncio when you write an HTTP client or server you have to construct bytes to 
write to the socket, and I'd be happy if
I could write b'HTTP/1.0 %d %b\r\n' % (status, message) rather than having to 
use str(status).encode('ascii') and
concatenation or join().


My own dbf module [1] would make use of this feature, and I'm sure some of the pdf modules would as well (I recall 
somebody chiming in about their own pdf module).


--
~Ethan~

[1] https://pypi.python.org/pypi/dbf
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Daniel Holth]

On Thu, Mar 27, 2014 at 2:53 PM, Guido van Rossum  wrote:
> So what's the use case for Python 2/3 compatible code? IMO the main use case
> for the PEP is simply to be able to construct bytes from a combination of a
> template and some input that may include further bytes and numbers. E.g. in
> asyncio when you write an HTTP client or server you have to construct bytes
> to write to the socket, and I'd be happy if I could write b'HTTP/1.0 %d
> %b\r\n' % (status, message) rather than having to use
> str(status).encode('ascii') and concatenation or join().

It seems to be notoriously difficult to understand or explain why
Unicode can still be very hard in Python 3 or in code that is in the
middle of being ported or has to run in both interpreters. As far as I
can tell part of it is when a symbol has type(str or bytes) depending
(declared as if we had a static type system with union types); some of
it is because incorrect mixing can happen without an exception, only
to be discovered later and far away in space and time from the error
(worse of all in a serialized file), and part of it is all of the not
easily checkable "types" a particular Unicode object has depending on
whether it contains surrogates or codes > n. Sometimes you might
simply disagree about whether an API should be returning bytes or
Unicode in mildly ambiguous cases like base64 encoding. Sometimes
Unicode is just intrinsically complicated.

For me this PEP holds the promise of being able to do work in the
bytes domain, with no accidental mixing ever, when I *really* want
bytes. For 2+3 I would get exceptions sometimes in Python 2 and
exceptions all the time in Python 3 for mistakes. I hope this is less
error prone in strict domains than for example u"string
processing".encode('latin1'). And I hope that there is very little
type(str or int) in HTTP for example or other "legitimate" bytes
domains but I don't know; I suspect that if you have a lot of problems
with bytes' %s then it's a clue you should use (u"%s" %
(argument)).encode() instead.

sprintf()'s version of %s just takes a char* and puts it in without
doing any type conversion of course. IANACL (I am not a C lawyer).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Glenn Linderman]

On 3/27/2014 11:59 AM, Guido van Rossum wrote:
On Thu, Mar 27, 2014 at 11:52 AM, Daniel Holth > wrote:


I feel not including %s is nuts. Should I write .replace('%b', '%s')?


I assume you meant .replace('%s', '%b') (unless you're converting 
Python 3 code to Python 2, which would mean you really are nuts :-).


But that's not going to help for the majority of code using %s -- as I 
am trying to argue, %s doesn't mean "expect the argument to be a str" 
and neither is that how it's commonly used (although it's *possible* 
that that is how *you* use it exclusively -- that doesn't make you 
nuts, just more strict than most people).


That _is_ how it is commonly used in Py2 when dealing with binary data 
in mixed ASCII/binary protocols, is what I've been hearing in this 
discussion, and what small use I've made of Py2 when some unported 
module forced me to use it (I started Python about the time Py3 was 
released)... the expected argument is a (Py2) str containing binary data 
(would be bytes in Py3).


While there are many other reasons to use %s in other coding situations, 
this is the only way to do bytes interpolations using %. And there is no 
%b in Py2, so for Py2/3 compatibility, %s needs to do bytes 
interpolations in Py3. And if it does, there is no need for %b in Py3 %, 
because they would be identical and redundant.



All I desperately need are APIs that provide enough unicode / str type
safety that I get an exception when mixing them accidentally... in my
own code, dynamic typing is usually a bug. As has been endlessly
discussed, %s for bytes is a bit like exposing sprintf()...


I don't understand that last claim (I can't figure out whether in this 
context is exposing sprintf() is considered good or bad). But apart 
from that, can you give some specific examples?


PS. I am not trying to be difficult. I honestly don't understand the 
use case yet, and the PEP doesn't do much to support it.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

On Thu, Mar 27, 2014 at 11:34 AM, Ethan Furman  wrote:

> On 03/27/2014 11:24 AM, Guido van Rossum wrote:
>
>> On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman wrote:
>>
>>>
>>> The biggest reason to use %s is to support a common code base for 2/3
>>> endeavors.
>>>
>>
>> But it's mostly useless for that purpose. In Python 2, in practice %s
>> doesn't mean "string". [...]
>>
>
> In Python 2 if one is using 'str' as a 'bytes' container, and doing
> interpolation, %s is the only choice available for other 'bytes' (aka other
> 'str's).  Note that I'm happy to be proven wrong on this point.  :)
>

That is true. And we can't change Python 2. I still have this idea in my
head that *most* cases where %s is used in Python 2 will break in Python 3
under the PEP's rules, but perhaps they are not the majority of situations
where the context is manipulating bytes. And I suppose that *very* few
internet protocols are designed to accept either an integer or the literal
string None, so that use case (which I brought up) isn't very realistic --
in fact it may be better to raise an exception rather than sending a
protocol violation.

So, I think you have changed my mind. I still like the idea of promoting %b
in pure Python 3 code to emphasize that it really behaves very differently
from %s; but I now have peace with %s as an alias. (It might also benefit
cases where somehow there's a symmetry in some Python 3 code between bytes
and str.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

On Thu, Mar 27, 2014 at 11:52 AM, Daniel Holth  wrote:

> I feel not including %s is nuts. Should I write .replace('%b', '%s')?
>

I assume you meant .replace('%s', '%b') (unless you're converting Python 3
code to Python 2, which would mean you really are nuts :-).

But that's not going to help for the majority of code using %s -- as I am
trying to argue, %s doesn't mean "expect the argument to be a str" and
neither is that how it's commonly used (although it's *possible* that that
is how *you* use it exclusively -- that doesn't make you nuts, just more
strict than most people).


> All I desperately need are APIs that provide enough unicode / str type
> safety that I get an exception when mixing them accidentally... in my
> own code, dynamic typing is usually a bug. As has been endlessly
> discussed, %s for bytes is a bit like exposing sprintf()...
>

I don't understand that last claim (I can't figure out whether in this
context is exposing sprintf() is considered good or bad). But apart from
that, can you give some specific examples?

PS. I am not trying to be difficult. I honestly don't understand the use
case yet, and the PEP doesn't do much to support it.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 11:24 AM, Guido van Rossum wrote:

On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman wrote:


The biggest reason to use %s is to support a common code base for 2/3 endeavors.


But it's mostly useless for that purpose. In Python 2, in practice %s doesn't mean 
"string". [...]


In Python 2 if one is using 'str' as a 'bytes' container, and doing interpolation, %s is the only choice available for 
other 'bytes' (aka other 'str's).  Note that I'm happy to be proven wrong on this point.  :)


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

So what's the use case for Python 2/3 compatible code? IMO the main use
case for the PEP is simply to be able to construct bytes from a combination
of a template and some input that may include further bytes and numbers.
E.g. in asyncio when you write an HTTP client or server you have to
construct bytes to write to the socket, and I'd be happy if I could write
b'HTTP/1.0 %d %b\r\n' % (status, message) rather than having to use
str(status).encode('ascii') and concatenation or join().


On Thu, Mar 27, 2014 at 11:47 AM, Brett Cannon  wrote:

>
>
> On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum 
> wrote:
>
>> Much better, but I'm still not happy with including %s at all. Otherwise
>> it's accept-worthy. (How's that for pressure. :-)
>>
>
> But if we only add %b and leave out %s then how is this going to lead to
> Python 2/3 compatible code since %b is not in Python 2? Or am I
> misunderstanding you?
>
> -Brett
>
>
>>
>>
>>  On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote:
>>
>>> On 03/27/2014 10:55 AM, Ethan Furman wrote:
>>>
 On 03/27/2014 10:29 AM, Guido van Rossum wrote:

>
> I also don't understand why we can't use %b instead of %s. AFAIK %b
> currently doesn't mean anything and I somehow don't
> expect we're likely to add it for other reasons (unless there's a
> proposal I'm missing?). Just like we use %a instead of
> %r to remind people that it's not quite the same (since it applies
> .encode('ascii', 'backslashreplace')), shouldn't we
> use anything *but* %s to remind people that that is also not the same
> (not at all, in fact)? The PEP's argument against
> %b ("rejected as not adding any value either in clarity or
> simplicity") is hardly a good reason.
>

 The biggest reason to use %s is to support a common code base for 2/3
 endeavors.  The biggest reason to not include %b
 is that it means binary number in format(); given that each type can
 invent it's own mini-language, this probably isn't
 a very strong argument against it.

 I have moderate feelings for keeping %s as a synonym for %b for
 backwards compatibility with Py2 code (when it's
 appropriate).

>>>
>>> Changed to:
>>> 
>>> --
>>> ``%b`` will insert a series of bytes.  These bytes are collected in one
>>> of two
>>> ways:
>>>
>>>   - input type supports ``Py_buffer`` [4]_?
>>>
>>> use it to collect the necessary bytes
>>>
>>>   - input type is something else?
>>> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
>>> ``TypeError``
>>>
>>> In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
>>> rejected
>>> as the string to bytes conversion requires an encoding, and we are
>>> refusing to
>>> guess; numbers are rejected because:
>>>
>>>   - what makes a number is fuzzy (float? Decimal? Fraction? some user
>>> type?)
>>>
>>>   - allowing numbers would lead to ambiguity between numbers and textual
>>> representations of numbers (3.14 vs '3.14')
>>>
>>>   - given the nature of wire formats, explicit is definitely better than
>>> implicit
>>>
>>> ``%s`` is included as a synonym for ``%b`` for the sole purpose of
>>> making 2/3 code
>>> bases easier to maintain.  Python 3 only code should use ``%b``.
>>>
>>> Examples::
>>>
>>> >>> b'%b' % b'abc'
>>> b'abc'
>>>
>>> >>> b'%b' % 'some string'.encode('utf8')
>>> b'some string'
>>>
>>> >>> b'%b' % 3.14
>>>
>>> Traceback (most recent call last):
>>> ...
>>> TypeError: b'%b' does not accept 'float'
>>>
>>> >>> b'%b' % 'hello world!'
>>>
>>> Traceback (most recent call last):
>>> ...
>>> TypeError: b'%b' does not accept 'str'
>>> 
>>> --
>>>
>>
>>>
>>> --
>>> ~Ethan~
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>>
>>  Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>>> guido%40python.org
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Daniel Holth]

I feel not including %s is nuts. Should I write .replace('%b', '%s')?
All I desperately need are APIs that provide enough unicode / str type
safety that I get an exception when mixing them accidentally... in my
own code, dynamic typing is usually a bug. As has been endlessly
discussed, %s for bytes is a bit like exposing sprintf()...

On Thu, Mar 27, 2014 at 2:41 PM, Guido van Rossum  wrote:
> Much better, but I'm still not happy with including %s at all. Otherwise
> it's accept-worthy. (How's that for pressure. :-)
>
>
> On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman  wrote:
>>
>> On 03/27/2014 10:55 AM, Ethan Furman wrote:
>>>
>>> On 03/27/2014 10:29 AM, Guido van Rossum wrote:


 I also don't understand why we can't use %b instead of %s. AFAIK %b
 currently doesn't mean anything and I somehow don't
 expect we're likely to add it for other reasons (unless there's a
 proposal I'm missing?). Just like we use %a instead of
 %r to remind people that it's not quite the same (since it applies
 .encode('ascii', 'backslashreplace')), shouldn't we
 use anything *but* %s to remind people that that is also not the same
 (not at all, in fact)? The PEP's argument against
 %b ("rejected as not adding any value either in clarity or simplicity")
 is hardly a good reason.
>>>
>>>
>>> The biggest reason to use %s is to support a common code base for 2/3
>>> endeavors.  The biggest reason to not include %b
>>> is that it means binary number in format(); given that each type can
>>> invent it's own mini-language, this probably isn't
>>> a very strong argument against it.
>>>
>>> I have moderate feelings for keeping %s as a synonym for %b for backwards
>>> compatibility with Py2 code (when it's
>>> appropriate).
>>
>>
>> Changed to:
>>
>> --
>> ``%b`` will insert a series of bytes.  These bytes are collected in one of
>> two
>> ways:
>>
>>   - input type supports ``Py_buffer`` [4]_?
>>
>> use it to collect the necessary bytes
>>
>>   - input type is something else?
>> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
>> ``TypeError``
>>
>> In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
>> rejected
>> as the string to bytes conversion requires an encoding, and we are
>> refusing to
>> guess; numbers are rejected because:
>>
>>   - what makes a number is fuzzy (float? Decimal? Fraction? some user
>> type?)
>>
>>   - allowing numbers would lead to ambiguity between numbers and textual
>> representations of numbers (3.14 vs '3.14')
>>
>>   - given the nature of wire formats, explicit is definitely better than
>> implicit
>>
>> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
>> 2/3 code
>> bases easier to maintain.  Python 3 only code should use ``%b``.
>>
>> Examples::
>>
>> >>> b'%b' % b'abc'
>> b'abc'
>>
>> >>> b'%b' % 'some string'.encode('utf8')
>> b'some string'
>>
>> >>> b'%b' % 3.14
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'float'
>>
>> >>> b'%b' % 'hello world!'
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'str'
>>
>> --
>>
>>
>> --
>> ~Ethan~
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Brett Cannon]

On Thu Mar 27 2014 at 2:42:40 PM, Guido van Rossum  wrote:

> Much better, but I'm still not happy with including %s at all. Otherwise
> it's accept-worthy. (How's that for pressure. :-)
>

But if we only add %b and leave out %s then how is this going to lead to
Python 2/3 compatible code since %b is not in Python 2? Or am I
misunderstanding you?

-Brett


>
>
> On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman  wrote:
>
>> On 03/27/2014 10:55 AM, Ethan Furman wrote:
>>
>>> On 03/27/2014 10:29 AM, Guido van Rossum wrote:
>>>

 I also don't understand why we can't use %b instead of %s. AFAIK %b
 currently doesn't mean anything and I somehow don't
 expect we're likely to add it for other reasons (unless there's a
 proposal I'm missing?). Just like we use %a instead of
 %r to remind people that it's not quite the same (since it applies
 .encode('ascii', 'backslashreplace')), shouldn't we
 use anything *but* %s to remind people that that is also not the same
 (not at all, in fact)? The PEP's argument against
 %b ("rejected as not adding any value either in clarity or simplicity")
 is hardly a good reason.

>>>
>>> The biggest reason to use %s is to support a common code base for 2/3
>>> endeavors.  The biggest reason to not include %b
>>> is that it means binary number in format(); given that each type can
>>> invent it's own mini-language, this probably isn't
>>> a very strong argument against it.
>>>
>>> I have moderate feelings for keeping %s as a synonym for %b for
>>> backwards compatibility with Py2 code (when it's
>>> appropriate).
>>>
>>
>> Changed to:
>> 
>> --
>> ``%b`` will insert a series of bytes.  These bytes are collected in one
>> of two
>> ways:
>>
>>   - input type supports ``Py_buffer`` [4]_?
>>
>> use it to collect the necessary bytes
>>
>>   - input type is something else?
>> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
>> ``TypeError``
>>
>> In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
>> rejected
>> as the string to bytes conversion requires an encoding, and we are
>> refusing to
>> guess; numbers are rejected because:
>>
>>   - what makes a number is fuzzy (float? Decimal? Fraction? some user
>> type?)
>>
>>   - allowing numbers would lead to ambiguity between numbers and textual
>> representations of numbers (3.14 vs '3.14')
>>
>>   - given the nature of wire formats, explicit is definitely better than
>> implicit
>>
>> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
>> 2/3 code
>> bases easier to maintain.  Python 3 only code should use ``%b``.
>>
>> Examples::
>>
>> >>> b'%b' % b'abc'
>> b'abc'
>>
>> >>> b'%b' % 'some string'.encode('utf8')
>> b'some string'
>>
>> >>> b'%b' % 3.14
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'float'
>>
>> >>> b'%b' % 'hello world!'
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'str'
>> 
>> --
>>
>
>>
>> --
>> ~Ethan~
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

Much better, but I'm still not happy with including %s at all. Otherwise
it's accept-worthy. (How's that for pressure. :-)


On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman  wrote:

> On 03/27/2014 10:55 AM, Ethan Furman wrote:
>
>> On 03/27/2014 10:29 AM, Guido van Rossum wrote:
>>
>>>
>>> I also don't understand why we can't use %b instead of %s. AFAIK %b
>>> currently doesn't mean anything and I somehow don't
>>> expect we're likely to add it for other reasons (unless there's a
>>> proposal I'm missing?). Just like we use %a instead of
>>> %r to remind people that it's not quite the same (since it applies
>>> .encode('ascii', 'backslashreplace')), shouldn't we
>>> use anything *but* %s to remind people that that is also not the same
>>> (not at all, in fact)? The PEP's argument against
>>> %b ("rejected as not adding any value either in clarity or simplicity")
>>> is hardly a good reason.
>>>
>>
>> The biggest reason to use %s is to support a common code base for 2/3
>> endeavors.  The biggest reason to not include %b
>> is that it means binary number in format(); given that each type can
>> invent it's own mini-language, this probably isn't
>> a very strong argument against it.
>>
>> I have moderate feelings for keeping %s as a synonym for %b for backwards
>> compatibility with Py2 code (when it's
>> appropriate).
>>
>
> Changed to:
> 
> --
> ``%b`` will insert a series of bytes.  These bytes are collected in one of
> two
> ways:
>
>   - input type supports ``Py_buffer`` [4]_?
>
> use it to collect the necessary bytes
>
>   - input type is something else?
> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
> ``TypeError``
>
> In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is
> rejected
> as the string to bytes conversion requires an encoding, and we are
> refusing to
> guess; numbers are rejected because:
>
>   - what makes a number is fuzzy (float? Decimal? Fraction? some user
> type?)
>
>   - allowing numbers would lead to ambiguity between numbers and textual
> representations of numbers (3.14 vs '3.14')
>
>   - given the nature of wire formats, explicit is definitely better than
> implicit
>
> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
> 2/3 code
> bases easier to maintain.  Python 3 only code should use ``%b``.
>
> Examples::
>
> >>> b'%b' % b'abc'
> b'abc'
>
> >>> b'%b' % 'some string'.encode('utf8')
> b'some string'
>
> >>> b'%b' % 3.14
>
> Traceback (most recent call last):
> ...
> TypeError: b'%b' does not accept 'float'
>
> >>> b'%b' % 'hello world!'
>
> Traceback (most recent call last):
> ...
> TypeError: b'%b' does not accept 'str'
> 
> --
>
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 10:55 AM, Ethan Furman wrote:

On 03/27/2014 10:29 AM, Guido van Rossum wrote:


I also don't understand why we can't use %b instead of %s. AFAIK %b currently 
doesn't mean anything and I somehow don't
expect we're likely to add it for other reasons (unless there's a proposal I'm 
missing?). Just like we use %a instead of
%r to remind people that it's not quite the same (since it applies 
.encode('ascii', 'backslashreplace')), shouldn't we
use anything *but* %s to remind people that that is also not the same (not at 
all, in fact)? The PEP's argument against
%b ("rejected as not adding any value either in clarity or simplicity") is 
hardly a good reason.


The biggest reason to use %s is to support a common code base for 2/3 
endeavors.  The biggest reason to not include %b
is that it means binary number in format(); given that each type can invent 
it's own mini-language, this probably isn't
a very strong argument against it.

I have moderate feelings for keeping %s as a synonym for %b for backwards 
compatibility with Py2 code (when it's
appropriate).


Changed to:
--
``%b`` will insert a series of bytes.  These bytes are collected in one of two
ways:

  - input type supports ``Py_buffer`` [4]_?
use it to collect the necessary bytes

  - input type is something else?
use its ``__bytes__`` method [5]_ ; if there isn't one, raise a 
``TypeError``

In particular, ``%b`` will not accept numbers nor ``str``.  ``str`` is rejected
as the string to bytes conversion requires an encoding, and we are refusing to
guess; numbers are rejected because:

  - what makes a number is fuzzy (float? Decimal? Fraction? some user type?)

  - allowing numbers would lead to ambiguity between numbers and textual
representations of numbers (3.14 vs '3.14')

  - given the nature of wire formats, explicit is definitely better than 
implicit

``%s`` is included as a synonym for ``%b`` for the sole purpose of making 2/3 
code
bases easier to maintain.  Python 3 only code should use ``%b``.

Examples::

>>> b'%b' % b'abc'
b'abc'

>>> b'%b' % 'some string'.encode('utf8')
b'some string'

>>> b'%b' % 3.14
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'float'

>>> b'%b' % 'hello world!'
Traceback (most recent call last):
...
TypeError: b'%b' does not accept 'str'
--

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

On Thu, Mar 27, 2014 at 10:55 AM, Ethan Furman  wrote:

> On 03/27/2014 10:29 AM, Guido van Rossum wrote:
>
>>
>> I also don't understand why we can't use %b instead of %s. AFAIK %b
>> currently doesn't mean anything and I somehow don't
>> expect we're likely to add it for other reasons (unless there's a
>> proposal I'm missing?). Just like we use %a instead of
>> %r to remind people that it's not quite the same (since it applies
>> .encode('ascii', 'backslashreplace')), shouldn't we
>> use anything *but* %s to remind people that that is also not the same
>> (not at all, in fact)? The PEP's argument against
>> %b ("rejected as not adding any value either in clarity or simplicity")
>> is hardly a good reason.
>>
>
> The biggest reason to use %s is to support a common code base for 2/3
> endeavors.


But it's mostly useless for that purpose. In Python 2, in practice %s
doesn't mean "string". It means "use the default formatting just as if I
was using print." And in theory it also means that -- in fact "call
__str__()" is the formal definition, and print is also defined as using
__str__, and this is all intentional. (I also intended __str__ to be
*mostly* the same as __repr__, with a specific exception for the str type
itself. In practice some frameworks have adopted a different
interpretation, making __repr__ produce something *more* "user friendly"
than __str__ but including newlines, because some people believe the main
use case for __repr__ is the interactive prompt. I believe this causes
problems for some *other* uses of __repr__, such as for producing an
"unambiguous" representation useful for e.g. logging -- but I don't want to
be too bitter about it. :-)

The biggest reason to not include %b is that it means binary number in
> format(); given that each type can invent it's own mini-language, this
> probably isn't a very strong argument against it.
>

Especially since I can't imagine the spelling in format() includes '%'.


> I have moderate feelings for keeping %s as a synonym for %b for backwards
> compatibility with Py2 code (when it's appropriate).
>

I think it's mere existence (with the restrictions currently in the PEP)
would cause more confusion than that is worth.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 10:29 AM, Guido van Rossum wrote:


I also don't understand why we can't use %b instead of %s. AFAIK %b currently 
doesn't mean anything and I somehow don't
expect we're likely to add it for other reasons (unless there's a proposal I'm 
missing?). Just like we use %a instead of
%r to remind people that it's not quite the same (since it applies 
.encode('ascii', 'backslashreplace')), shouldn't we
use anything *but* %s to remind people that that is also not the same (not at 
all, in fact)? The PEP's argument against
%b ("rejected as not adding any value either in clarity or simplicity") is 
hardly a good reason.


The biggest reason to use %s is to support a common code base for 2/3 endeavors.  The biggest reason to not include %b 
is that it means binary number in format(); given that each type can invent it's own mini-language, this probably isn't 
a very strong argument against it.


I have moderate feelings for keeping %s as a synonym for %b for backwards compatibility with Py2 code (when it's 
appropriate).


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

Wow. I'm pretty consistent. I still like that. :-)


On Thu, Mar 27, 2014 at 10:31 AM, MRAB  wrote:

> On 2014-03-27 15:58, Ethan Furman wrote:
>
>> On 03/27/2014 04:42 AM, Nick Coghlan wrote:
>>
>>>
>>> I also seem to recall Guido saying he liked it [%a], which flipped the
>>> discussion from "do we have a good rationale for including it?" to "do
>>> we have a good rationale for the BDFL to ignore his instincts?".
>>> However, it would be up to Guido to confirm that recollection, and if
>>> "Guido likes it" is part of the reason for inclusion of the %a code,
>>> the PEP should mention that explicitly.
>>>
>>
>> I checked Guido's posts (Subject contains PEP 461, From contains guido)
>> and did not see anything to that effect.
>>
>>  Date: Mon, 13 Jan 2014 12:09:23 -0800
> Subject: Re: [Python-Dev] PEP 460 reboot
>
> """If we have %b for strictly interpolating bytes, I'm fine with adding
> %a for calling ascii() on the argument and then interpolating the
> result after ASCII-encoding it."""
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[MRAB]

On 2014-03-27 15:58, Ethan Furman wrote:

On 03/27/2014 04:42 AM, Nick Coghlan wrote:


I also seem to recall Guido saying he liked it [%a], which flipped the
discussion from "do we have a good rationale for including it?" to "do
we have a good rationale for the BDFL to ignore his instincts?".
However, it would be up to Guido to confirm that recollection, and if
"Guido likes it" is part of the reason for inclusion of the %a code,
the PEP should mention that explicitly.


I checked Guido's posts (Subject contains PEP 461, From contains guido) and did 
not see anything to that effect.


Date: Mon, 13 Jan 2014 12:09:23 -0800
Subject: Re: [Python-Dev] PEP 460 reboot

"""If we have %b for strictly interpolating bytes, I'm fine with adding
%a for calling ascii() on the argument and then interpolating the
result after ASCII-encoding it."""

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

I also don't understand why we can't use %b instead of %s. AFAIK %b
currently doesn't mean anything and I somehow don't expect we're likely to
add it for other reasons (unless there's a proposal I'm missing?). Just
like we use %a instead of %r to remind people that it's not quite the same
(since it applies .encode('ascii', 'backslashreplace')), shouldn't we use
anything *but* %s to remind people that that is also not the same (not at
all, in fact)? The PEP's argument against %b ("rejected as not adding any
value either in clarity or simplicity") is hardly a good reason.


On Thu, Mar 27, 2014 at 10:20 AM, Guido van Rossum  wrote:

> Actually, I had ignored this discussion for so long that I was surprised
> by the outcome. My main use case isn't printing a number that may already
> be a string (I understand why that isn't reasonable when the output is
> expected to be bytes); it's printing a usually numeric value that may
> sometimes be None. It's a little surprising to have to use %a for this, but
> I guess I can live with it.
>
>
> On Thu, Mar 27, 2014 at 8:58 AM, Ethan Furman  wrote:
>
>> On 03/27/2014 04:42 AM, Nick Coghlan wrote:
>>
>>>
>>> I also seem to recall Guido saying he liked it [%a], which flipped the
>>>
>>> discussion from "do we have a good rationale for including it?" to "do
>>> we have a good rationale for the BDFL to ignore his instincts?".
>>> However, it would be up to Guido to confirm that recollection, and if
>>> "Guido likes it" is part of the reason for inclusion of the %a code,
>>> the PEP should mention that explicitly.
>>>
>>
>> I checked Guido's posts (Subject contains PEP 461, From contains guido)
>> and did not see anything to that effect.
>>
>> --
>> ~Ethan~
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Guido van Rossum]

Actually, I had ignored this discussion for so long that I was surprised by
the outcome. My main use case isn't printing a number that may already be a
string (I understand why that isn't reasonable when the output is expected
to be bytes); it's printing a usually numeric value that may sometimes be
None. It's a little surprising to have to use %a for this, but I guess I
can live with it.


On Thu, Mar 27, 2014 at 8:58 AM, Ethan Furman  wrote:

> On 03/27/2014 04:42 AM, Nick Coghlan wrote:
>
>>
>> I also seem to recall Guido saying he liked it [%a], which flipped the
>>
>> discussion from "do we have a good rationale for including it?" to "do
>> we have a good rationale for the BDFL to ignore his instincts?".
>> However, it would be up to Guido to confirm that recollection, and if
>> "Guido likes it" is part of the reason for inclusion of the %a code,
>> the PEP should mention that explicitly.
>>
>
> I checked Guido's posts (Subject contains PEP 461, From contains guido)
> and did not see anything to that effect.
>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] multiprocessing BaseManager's "serializer" argument

[Yuriy Taraday]

On Thu, Mar 27, 2014 at 8:42 PM, Brett Cannon  wrote:

> Creating the bug is the best bet as then the reason can be discussed
> there. If there are no objections then you can create a patch to make sure
> it's tested and documented.


Created http://bugs.python.org/issue21078

-- 

Kind regards, Yuriy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes #20668

[Antoine Pitrou]

On Thu, 27 Mar 2014 17:22:01 +0100 (CET)
yury.selivanov  wrote:
> http://hg.python.org/cpython/rev/bcc77493249c
> changeset:   89995:bcc77493249c
> user:Yury Selivanov 
> date:Thu Mar 27 12:21:20 2014 -0400
> summary:
>   asyncio.tests: Autodiscover asyncio tests. Patch by Vajrasky Kok. Closes 
> #20668
> 
> files:
>   Lib/test/test_asyncio/__init__.py |  22 --
>   Misc/NEWS |   3 ++
>   2 files changed, 13 insertions(+), 12 deletions(-)

Perhaps the tests.txt file should be removed from the repo?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] multiprocessing BaseManager's "serializer" argument

[Brett Cannon]

Creating the bug is the best bet as then the reason can be discussed there.
If there are no objections then you can create a patch to make sure it's
tested and documented.

On Thu Mar 27 2014 at 12:20:00 PM, Yuriy Taraday 
wrote:

> Hello.
>
> Is there any reason BaseManager's "serializer" argument has never been
> documented?
> We're going to use BaseManager for simple secure local RPC and for the
> "secure" part we can't use pickle. Can we hope that argument won't go away?
> Should I create a bug and provide some patch for the docs to add mention of
> this argument?
>
> --
>
> Kind regards, Yuriy.
>  ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: inspect.Signature: Add 'Signature.from_callable' classmethod. Closes #17373

[Antoine Pitrou]

On Thu, 27 Mar 2014 17:12:02 +0100 (CET)
yury.selivanov  wrote:
>  
> +.. classmethod:: Signature.from_callable(obj)
> +
> +   Return a :class:`Signature` (or its subclass) object for a given 
> callable
> +   ``obj``. This method simplifies subclassing of :class:`Signature`:
> +
> +   ::
> +
> + class MySignature(Signature):
> + pass
> + sig = MySignature.from_callable(min)
> + assert isinstance(sig, MySignature)
> +

This needs a "versionadded" tag.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 04:42 AM, Nick Coghlan wrote:


I also seem to recall Guido saying he liked it [%a], which flipped the
discussion from "do we have a good rationale for including it?" to "do
we have a good rationale for the BDFL to ignore his instincts?".
However, it would be up to Guido to confirm that recollection, and if
"Guido likes it" is part of the reason for inclusion of the %a code,
the PEP should mention that explicitly.


I checked Guido's posts (Subject contains PEP 461, From contains guido) and did 
not see anything to that effect.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] multiprocessing BaseManager's "serializer" argument

[Yuriy Taraday]

Hello.

Is there any reason BaseManager's "serializer" argument has never been
documented?
We're going to use BaseManager for simple secure local RPC and for the
"secure" part we can't use pickle. Can we hope that argument won't go away?
Should I create a bug and provide some patch for the docs to add mention of
this argument?

-- 

Kind regards, Yuriy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Ethan Furman]

On 03/27/2014 04:24 AM, Antoine Pitrou wrote:

On Tue, 25 Mar 2014 15:37:11 -0700 Ethan Furman wrote:


``%a`` will call ``ascii()`` on the interpolated value.  This is intended
as a debugging aid, rather than something that should be used in production.
Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
representation.  Use cases include developing a new protocol and writing
landmarks into the stream; debugging data going into an existing protocol
to see if the problem is the protocol itself or bad data; a fall-back for a
serialization format; or even a rudimentary serialization format when
defining ``__bytes__`` would not be appropriate [8].


The "use cases" you are enumerating for "%a" are chimeric.


Cool word!  Haven't seen it a long time.  :)


Did you *actually* do those things in real life, or are you inventing them
for the PEP?


The examples came from Jim Jewett, but I can easily see myself using them.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements

[Guido van Rossum]

BTW, my intention was also to set a precedent for future additions to the
list. "We did this before" is a great argument to help smooth the path,
without bypassing deliberations altogether.
On Mar 27, 2014 2:33 AM, "Nick Coghlan"  wrote:

> On 27 March 2014 18:02, Stephen J. Turnbull  wrote:
> > Alex Gaynor writes:
> >
> >  > Here's my proposed list of such featuers:
> >
> > And suppose that list grows over time?  After all, it once was [].
> >
> > If we go for a feature-by-feature list, that has two more-or-less
> > hidden costs.  (1) Python-Dev has to specify which ones, and either
> > risks a new specification debate in the future, or needs to spend
> > time now describing criteria and processes for extending the list.
>
> It's not a hidden cost - it's a deliberately chosen one. Guido was
> wary of an open-ended agreement, so by enumerating the precise set of
> missing features in Python 2.7 that are causing concern for the
> network security folks, we get to address the immediate problem,
> without granting permission to backport further arbitrary features
> without additional discussion.
>
> > (2) Users may need to worry about the list.  (OTOH, as long as the
> > list is restricted to features in certain modules, users can choose to
> > assume anything in those modules may have changed behavior and that's
> > no different from Nick's proposal for them.)
>
> The PEP already specifically advises that cross-version compatible
> code use feature detection rather than version checks. For network
> security, it's recommended to avoid using the low level modules
> directly, anyway - it's much better to use a higher level library like
> requests, to reduce the number of places where we need to get the
> security design right.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Thu, 27 Mar 2014 08:50:01 -0700
Daniel Stutzbach  wrote:
> 
> Due to way the heapq is implemented, it can't provide an efficient API for
> removing an arbitrary item.  Swapping with the last element allows you to
> efficiently remove the item at a particular index, but you first need to
> find the current index of an item, which requires a O(n) scan.
> 
> To provide efficient cancellation and removal, a heap implementation needs
> some way to efficiently answer "What is the current index of this item?".
>  There are a couple of ways to achieve that, but they all require more
> storage than heapq's list-based approach.

You are right. I was assuming the index is already known.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Guido van Rossum]

Surely you can show empathy and still explain why it's not that easy.
On Mar 27, 2014 2:11 AM, "Maciej Fijalkowski"  wrote:

> On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore  wrote:
> > On 27 March 2014 08:16, Maciej Fijalkowski  wrote:
> >> And random pieces of C included in the standard library can be
> >> shuffled under the carpet under the disguise of upgrade or what are
> >> you suggesting?
> >
> > The sort of thing that happens is that the relevant approvers will
> > accept python-dev as a "trusted supplier" and then Python upgrades are
> > acceptable subject to review of the changes, etc. For a new module,
> > there is a whole other level of questions around how do we trust the
> > person who developed the code, do we need to do a full code review,
> > etc?
> >
> > It's a bit unfair to describe the process as "random pieces of C"
> > being "shuffled under the carpet". (Although there probably are
> > environments where that is uncomfortably close to the truth :-()
> >
> > Paul
>
> I just find "my company is stupid so let's work around it by putting
> stuff to python standard library" unacceptable argument for python-dev
> and all the python community.
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Marko Rauhamaa]

Thomas Wouters :

> Not to mention discussion about whether it shouldn't just be an existing
> PyPI package, like http://pypi.python.org/pypi/blist, rather than a new
> implementation.

I'm fine with any implementation as long as it is in the standard
library.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[R. David Murray]

On Thu, 27 Mar 2014 12:24:49 +0100, Antoine Pitrou  wrote:
> On Tue, 25 Mar 2014 15:37:11 -0700
> Ethan Furman  wrote:
> > 
> > ``%a`` will call ``ascii()`` on the interpolated value.  This is intended
> > as a debugging aid, rather than something that should be used in production.
> > Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
> > representation.  Use cases include developing a new protocol and writing
> > landmarks into the stream; debugging data going into an existing protocol
> > to see if the problem is the protocol itself or bad data; a fall-back for a
> > serialization format; or even a rudimentary serialization format when
> > defining ``__bytes__`` would not be appropriate [8].
> 
> The "use cases" you are enumerating for "%a" are chimeric. Did you
> *actually* do those things in real life, or are you inventing them for
> the PEP?

The use cases came from someone else (Jim Jewett?) so you should
be asking him, not Ethan :)

As for the "did you actually do those things in real life", I know I've
done the "dump the repr into the data (protocol) stream to see what
I've really got here" debug trick in the string context, so I have no
doubt that I will want to do it in the bytes context as well.  In fact,
it is probably somewhat more likely in the bytes context, since I know
I've been in situations with data exchange protocols where I couldn't
get console output and setting up logging was much more painful than
just dumping the debug data into into the data stream.  Or where doing
so made it much clearer what was going on than separate logging would.
I've done the 'landmark' thing as well, in the string context; that can be
very useful when doing incremental test driven development.  (Granted, you
could do that with __bytes__; you might well be writing a __bytes__
method anyway as the next step, but it *is* more overhead/boilerplate than
just starting with %a...and it gets people used to reaching for __bytes__
for the "wrong" purpose, which is Nick's concern).  In theory I can see
using %a for serialization in certain limited contexts (I've done that
with string repr in private utility scripts), but in practice I doubt
that would happen in a binary context, since those are much more likely
to be actually going over a "wire" of some sort (ie: places you really
don't want to use eval even when it would work).

So yeah, I think %a has *practical* utility.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Thomas Wouters]

On Wed, Mar 26, 2014 at 9:55 PM, Benjamin Peterson wrote:

> On Wed, Mar 26, 2014, at 13:31, Marko Rauhamaa wrote:
> >
> > I have made a full implementation of a balanced tree and would like to
> > know what the process is to have it considered for inclusion in Python
> > 3.
>
> It's not a bad idea. (I believe others have proposed an red-black tree.)
> Certainly, it requires a PEP and a few months of bikesheding, though.


Not to mention discussion about whether it shouldn't just be an existing
PyPI package, like http://pypi.python.org/pypi/blist, rather than a new
implementation.

-- 
Thomas Wouters 

Hi! I'm an email virus! Think twice before sending your email to help me
spread!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Serhiy Storchaka]

27.03.14 00:16, Guido van Rossum написав(ла):

Yeah, so the pyftp fix is to keep track of how many timers were
cancelled, and if the number exceeds a threshold it just recreates the
heap, something like

heap = [x for x in heap if not x.cancelled]
heapify(heap)


See also http://bugs.python.org/issue13451 which proposes such approach 
for the sched module.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Nick Coghlan]

On 27 March 2014 21:24, Antoine Pitrou  wrote:
> On Tue, 25 Mar 2014 15:37:11 -0700
> Ethan Furman  wrote:
>>
>> ``%a`` will call ``ascii()`` on the interpolated value.  This is intended
>> as a debugging aid, rather than something that should be used in production.
>> Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
>> representation.  Use cases include developing a new protocol and writing
>> landmarks into the stream; debugging data going into an existing protocol
>> to see if the problem is the protocol itself or bad data; a fall-back for a
>> serialization format; or even a rudimentary serialization format when
>> defining ``__bytes__`` would not be appropriate [8].
>
> The "use cases" you are enumerating for "%a" are chimeric. Did you
> *actually* do those things in real life, or are you inventing them for
> the PEP?

I'm the one that raised the "discourage misuse of __bytes__" concern,
so I'd like %a to stay in at least for that reason. %a is a perfectly
well defined format code (albeit one you'd only be likely to use while
messing about with serialisation protocols, as the PEP describes - for
example, if a %b code was ending up producing wrong data, you might
switch to %a temporarily to get a better idea of where the bad data
was coming from), while using __bytes__ to make %s behave the way %a
is defined in the PEP would just be wrong in most cases. I consider %a
the preemptive PEP 308 of binary interpolation format codes - in the
absence of %a, I'm certain that users would end up abusing __bytes__
and %s to get the same effect, just as they used the known bug magnet
that was the and/or hack for a long time in the absence of PEP 308.

I also seem to recall Guido saying he liked it, which flipped the
discussion from "do we have a good rationale for including it?" to "do
we have a good rationale for the BDFL to ignore his instincts?".
However, it would be up to Guido to confirm that recollection, and if
"Guido likes it" is part of the reason for inclusion of the %a code,
the PEP should mention that explicitly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Antoine Pitrou]

On Thu, 27 Mar 2014 09:25:05 +
Kristján Valur Jónsson  wrote:
> True.
> I've long since added a heapdel() to our local fork.
> a heappop(idx=0) extension would do the same
> I can provide a patch if there is interest.

I think either of them would be cool. I don't know it would be approved
by Raymond or whoever else.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Nick Coghlan]

On 27 March 2014 20:47, Victor Stinner  wrote:
> The PEP 461 looks good to me. It's a nice addition to Python 3.5 and
> the PEP is well defined.

+1 from me as well. One minor request is that I don't think the
rationale for rejecting numbers from "%s" is incomplete - IIRC, the
problem there is that the normal path for handling those is the
coercion via str() and this proposal deliberately *doesn't* allow that
path. That means supporting numbers would mean writing a lot of
*additional* code, and that isn't needed since 2/3 compatible code can
just be adjusted to use an appropriate numeric code.

> Note: I fixed a typo in your PEP (reST syntax).

I also committed a couple of markup tweaks, since it seemed easier to
just fix them than explain what was broken. However, there are also
two dead footnotes (4 & 5), which I have left alone - I'm not sure if
the problem is a missing reference, or if the footnote can go away
now.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Antoine Pitrou]

On Tue, 25 Mar 2014 15:37:11 -0700
Ethan Furman  wrote:
> 
> ``%a`` will call ``ascii()`` on the interpolated value.  This is intended
> as a debugging aid, rather than something that should be used in production.
> Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
> representation.  Use cases include developing a new protocol and writing
> landmarks into the stream; debugging data going into an existing protocol
> to see if the problem is the protocol itself or bad data; a fall-back for a
> serialization format; or even a rudimentary serialization format when
> defining ``__bytes__`` would not be appropriate [8].

The "use cases" you are enumerating for "%a" are chimeric. Did you
*actually* do those things in real life, or are you inventing them for
the PEP?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Chris Angelico]

On Thu, Mar 27, 2014 at 10:10 PM, Nick Coghlan  wrote:
> Extending our trust to include a new component isn't to be done
> lightly, but it *does* genuinely save work in the long run for a whole
> lot of other people whenever we choose to do so, and that is the point
> Stephen was making.

Exactly so.

Blessing a module as part of the stdlib is an important mark of trust,
and one that is most definitely not stupid to respect. (In case my
previous message was ambiguous, I am fully in favour of people
trusting python.org in this way, and fully understanding of people NOT
trusting other sources, or at least not automatically.)

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Nick Coghlan]

On 27 March 2014 20:38, Chris Angelico  wrote:
> On Thu, Mar 27, 2014 at 8:58 PM, Nick Coghlan  wrote:
>>On 27 March 2014 19:10, Maciej Fijalkowski  wrote:
>>> I just find "my company is stupid so let's work around it by putting
>>> stuff to python standard library" unacceptable argument for python-dev
>>> and all the python community.
>>
>> Due diligence and prudent risk management are not stupid - most open
>> source projects and small companies just don't have the luxury of
>> worrying about them, as they're so far down the list of concerns that
>> the additional risk of using arbitrary code downloaded off the
>> internet doesn't even register.
>
> I don't think anyone's saying it's stupid to be cautious, but more
> that it's stupid to blindly accept the latest python.org release and
> *not* accept something from another source. And if that's stupid,
> well, I'm stupid too - blindly accepting a whole lot of binary package
> updates because they're on ftp.au.debian.org, for instance. Why do I
> trust that, and not random sites on the internet? Because I trust that
> the Debian package maintainers to check what goes through, and I trust
> that there are people with reputations at stake, who won't want to
> send something dodgy through. It's not perfect, but it's a whole lot
> easier than checking every single package that goes through.

Right - trusting the PSF and the Debian package review process are
reasonable trade-offs at a personal level. However, it highlights why
this is still a benefit of bringing things into the standard library:
a lot of the overhead in review and audit processes is incurred based
on the *number of components to be reviewed*, rather than the amount
of code they contain. Other aspects of the overhead are incurred *per
organisation trusted*.

Inclusion in the standard library means not only bringing something
under the PSF's "trust umbrella", but also within the core Python
component. While we usually use corporate environments as our example,
it applies equally to Linux distros - they generally have Python
packaged already, and will eventually inherit standard library
improvements. For third party packages, it requires someone to do the
work to get them installed.

Extending our trust to include a new component isn't to be done
lightly, but it *does* genuinely save work in the long run for a whole
lot of other people whenever we choose to do so, and that is the point
Stephen was making.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3

[Victor Stinner]

The PEP 461 looks good to me. It's a nice addition to Python 3.5 and
the PEP is well defined.

I can help to implement it. Maybe, it would be nice to provide an
implementation as a third-party party module on PyPI for Python
2.6-3.4.

Note: I fixed a typo in your PEP (reST syntax).

Victor

2014-03-26 23:47 GMT+01:00 Ethan Furman :
>> This one is wrong:
>>
> repr(b'abc').encode('ascii', 'backslashreplace')
>>
>> b"b'abc'"
>
>
> Fixed, thanks.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Chris Angelico]

On Thu, Mar 27, 2014 at 8:58 PM, Nick Coghlan  wrote:
>On 27 March 2014 19:10, Maciej Fijalkowski  wrote:
>> I just find "my company is stupid so let's work around it by putting
>> stuff to python standard library" unacceptable argument for python-dev
>> and all the python community.
>
> Due diligence and prudent risk management are not stupid - most open
> source projects and small companies just don't have the luxury of
> worrying about them, as they're so far down the list of concerns that
> the additional risk of using arbitrary code downloaded off the
> internet doesn't even register.

I don't think anyone's saying it's stupid to be cautious, but more
that it's stupid to blindly accept the latest python.org release and
*not* accept something from another source. And if that's stupid,
well, I'm stupid too - blindly accepting a whole lot of binary package
updates because they're on ftp.au.debian.org, for instance. Why do I
trust that, and not random sites on the internet? Because I trust that
the Debian package maintainers to check what goes through, and I trust
that there are people with reputations at stake, who won't want to
send something dodgy through. It's not perfect, but it's a whole lot
easier than checking every single package that goes through.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Nick Coghlan]

On 27 March 2014 12:11, Eli Bendersky  wrote:
> On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson 
> wrote:
>> I'm not sure if that's a good thing or not.
>
> YMMV but IMHO this is a good thing. PEPs provide a single point of reference
> to a discussion that would otherwise be spread over multiple centi-threads
> (not that PEPs don't create centi-threads, but they outlive them in a way).

>From my point of view, the primary purpose of the PEP process is to
provide a way for us to finally say "yes" to controversial proposals
that have valid arguments against them. When things are obviously good
ideas that don't impose a big maintenance burden, nobody really
objects if we skip the PEP process (that isn't always a good thing -
directory and zipfile execution flew under the radar for years because
it was such a neat idea that Guido approved it directly in the issue,
and then we forgot to mention it in the 2.6 What's New).

Some ideas aren't obviously good, or a suitable API isn't obvious, or
they impose a major additional maintenance burden, or they require a
change to our development policies. In those cases, the PEP process
allows us to collectively ask the question "Is this worth the
hassle?". Cases like the restoration of binary interpolation support,
or my proposal to backport network security features, also showcase
how the PEP process can be used to refine the *question* so the PEP
champion is forced to figure out what they *really* want, and propose
a solution that clearly solves that specific problem, rather than
overreaching and asking for more than is needed. (This is also
reflected in the relative fates of the current matrix multiplication
proposal and previous more general proposals)

With the introduction of the BDFL-Delegate system, and then the
decision last year to give the "Discussions-To" header a bit more
force and allow groups like the Python Packaging Authority to make use
of the PEP process independently of python-dev, the PEP process is
also becoming more streamlined, making it more effective in its role
as a tool for establishing consensus - there's less need to convince
someone to drop a veto without a PEP, as the PEP process itself is
getting less painful.

Most of the time when I hear people say "the PEP process is too
difficult", I eventually find that what they really mean is "learning
the kinds of things that python-dev are likely to be worried about,
and ensuring that the PEP adequately addresses their concerns, and
listening to feedback, and reconsidering what I actually want, and
revising my proposal, such that they eventually say yes is too time
consuming".

Helping people to learn exactly how to navigate that process is
actually one of the main roles of python-ideas these days, although we
don't do a good job (at all) of advertising that fact.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Nick Coghlan]

On 27 March 2014 19:10, Maciej Fijalkowski  wrote:
> On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore  wrote:
>> On 27 March 2014 08:16, Maciej Fijalkowski  wrote:
>>> And random pieces of C included in the standard library can be
>>> shuffled under the carpet under the disguise of upgrade or what are
>>> you suggesting?
>>
>> The sort of thing that happens is that the relevant approvers will
>> accept python-dev as a "trusted supplier" and then Python upgrades are
>> acceptable subject to review of the changes, etc. For a new module,
>> there is a whole other level of questions around how do we trust the
>> person who developed the code, do we need to do a full code review,
>> etc?
>>
>> It's a bit unfair to describe the process as "random pieces of C"
>> being "shuffled under the carpet". (Although there probably are
>> environments where that is uncomfortably close to the truth :-()
>>
>> Paul
>
> I just find "my company is stupid so let's work around it by putting
> stuff to python standard library" unacceptable argument for python-dev
> and all the python community.

Due diligence and prudent risk management are not stupid - most open
source projects and small companies just don't have the luxury of
worrying about them, as they're so far down the list of concerns that
the additional risk of using arbitrary code downloaded off the
internet doesn't even register.

As organisations get larger, they have more to lose, so they typically
start worrying about that kind of thing. Properly vetting software for
licensing and potential security issues is expensive though, so they
usually prefer to outsource that task to trusted providers (this is
one of the key concepts that gets me paid). Contractual arrangements
and brand reputations start to replace blind trust in upstream
developers.

Is this less efficient than full open collaboration? Yes, it is - the
"verify" part of "trust, but verify" comes at a real cost in time and
money. However, there are plenty of good reasons that phrase is
"trust, but verify" rather than "trust unconditionally".

You may choose not to care about those more cautious users, and that's
fine - but they're still worth taking into account when making design
decisions if you want to cross the marketing chasm from "early
adopters" to everyone else.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Nick Coghlan]

On 27 March 2014 18:11, Stephen J. Turnbull  wrote:
> Nick Coghlan writes:
>
>  >  On 27 Mar 2014 07:02, "Guido van Rossum"  wrote:
>  >> Actually, the first step is publish it on PyPI, the second is to
>  >> get a fair number of happy users there. The bar for getting something
>  >> included into the stdlib is pretty high
>
>  > The "why not a third party module?" bar also got a fair bit higher
>  > with Python 3.4 - by bundling pip, we have deliberately made third
>  > party modules easier to consume, thus weakening the convenience
>  > argument that applies to stdlib inclusion.
>
> Maybe.  That depends on if you care about the convenience of folks who
> have to get new modules past Corporate Security, but it's easier to
> get an upgrade of the whole shebang.  I don't think it's ever really
> been resolved whether they're a "typical case that won't go away" or a
> special group whose special needs should be considered.

I'm all too well aware of this particular problem (although thankfully
don't have to deal with it myself any more), and it's a key part of
why I said "a fair bit higher" rather than "insurmountable" :)

Such environments *do* usually have procedures to get additional open
source software approved, it's just a pain. That pain then becomes
another factor to take into account deciding between using a simpler
solution, rolling your own custom solution, or tackling your
organisation's module approval process.

That said, getting approval is definitely easier when the request is
to trust "the Python Software Foundation" vs "J. Random Programmer's
account on GitHub", so yes, it still counts in favour of sufficiently
compelling stdlib additions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements

[Nick Coghlan]

On 27 March 2014 18:02, Stephen J. Turnbull  wrote:
> Alex Gaynor writes:
>
>  > Here's my proposed list of such featuers:
>
> And suppose that list grows over time?  After all, it once was [].
>
> If we go for a feature-by-feature list, that has two more-or-less
> hidden costs.  (1) Python-Dev has to specify which ones, and either
> risks a new specification debate in the future, or needs to spend
> time now describing criteria and processes for extending the list.

It's not a hidden cost - it's a deliberately chosen one. Guido was
wary of an open-ended agreement, so by enumerating the precise set of
missing features in Python 2.7 that are causing concern for the
network security folks, we get to address the immediate problem,
without granting permission to backport further arbitrary features
without additional discussion.

> (2) Users may need to worry about the list.  (OTOH, as long as the
> list is restricted to features in certain modules, users can choose to
> assume anything in those modules may have changed behavior and that's
> no different from Nick's proposal for them.)

The PEP already specifically advises that cross-version compatible
code use feature detection rather than version checks. For network
security, it's recommended to avoid using the low level modules
directly, anyway - it's much better to use a higher level library like
requests, to reduce the number of places where we need to get the
security design right.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Kristján Valur Jónsson]

True.
I've long since added a heapdel() to our local fork.
a heappop(idx=0) extension would do the same
I can provide a patch if there is interest.
K


Ideally, I think you should be able to replace the cancelled item with
the last item in the heap and then fix the heap in logarithmic time,
but the heapq API doesn't seem to provide a way to do this.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Maciej Fijalkowski]

On Thu, Mar 27, 2014 at 11:07 AM, Paul Moore  wrote:
> On 27 March 2014 08:16, Maciej Fijalkowski  wrote:
>> And random pieces of C included in the standard library can be
>> shuffled under the carpet under the disguise of upgrade or what are
>> you suggesting?
>
> The sort of thing that happens is that the relevant approvers will
> accept python-dev as a "trusted supplier" and then Python upgrades are
> acceptable subject to review of the changes, etc. For a new module,
> there is a whole other level of questions around how do we trust the
> person who developed the code, do we need to do a full code review,
> etc?
>
> It's a bit unfair to describe the process as "random pieces of C"
> being "shuffled under the carpet". (Although there probably are
> environments where that is uncomfortably close to the truth :-()
>
> Paul

I just find "my company is stupid so let's work around it by putting
stuff to python standard library" unacceptable argument for python-dev
and all the python community.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Paul Moore]

On 27 March 2014 08:16, Maciej Fijalkowski  wrote:
> And random pieces of C included in the standard library can be
> shuffled under the carpet under the disguise of upgrade or what are
> you suggesting?

The sort of thing that happens is that the relevant approvers will
accept python-dev as a "trusted supplier" and then Python upgrades are
acceptable subject to review of the changes, etc. For a new module,
there is a whole other level of questions around how do we trust the
person who developed the code, do we need to do a full code review,
etc?

It's a bit unfair to describe the process as "random pieces of C"
being "shuffled under the carpet". (Although there probably are
environments where that is uncomfortably close to the truth :-()

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] On the necessity of PEPs [was "collections.sortedtree"]

[Stephen J. Turnbull]

Eli Bendersky writes:
 > On Wed, Mar 26, 2014 at 2:27 PM, Benjamin Peterson  
 > wrote:

 >> I would have said that, too, several years ago, but I think we've
 >> been requiring (or using anyway) PEPs for a lot more things now.

 > YMMV but IMHO this is a good thing.

FWIW I was just talking to Matz yesterday, and asked him about it.  He
said he thought PEPs worked great for Python (in the context of
explain that his (and Ruby's) style is different).

Just-ask-the-nearest-independent-opinion-ly y'rs,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Maciej Fijalkowski]

On Thu, Mar 27, 2014 at 10:11 AM, Stephen J. Turnbull
 wrote:
> Nick Coghlan writes:
>
>  >  On 27 Mar 2014 07:02, "Guido van Rossum"  wrote:
>  >> Actually, the first step is publish it on PyPI, the second is to
>  >> get a fair number of happy users there. The bar for getting something
>  >> included into the stdlib is pretty high
>
>  > The "why not a third party module?" bar also got a fair bit higher
>  > with Python 3.4 - by bundling pip, we have deliberately made third
>  > party modules easier to consume, thus weakening the convenience
>  > argument that applies to stdlib inclusion.
>
> Maybe.  That depends on if you care about the convenience of folks who
> have to get new modules past Corporate Security, but it's easier to
> get an upgrade of the whole shebang.  I don't think it's ever really
> been resolved whether they're a "typical case that won't go away" or a
> special group whose special needs should be considered.
>
> Steve

And random pieces of C included in the standard library can be
shuffled under the carpet under the disguise of upgrade or what are
you suggesting?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] collections.sortedtree

[Stephen J. Turnbull]

Nick Coghlan writes:

 >  On 27 Mar 2014 07:02, "Guido van Rossum"  wrote:
 >> Actually, the first step is publish it on PyPI, the second is to
 >> get a fair number of happy users there. The bar for getting something
 >> included into the stdlib is pretty high

 > The "why not a third party module?" bar also got a fair bit higher
 > with Python 3.4 - by bundling pip, we have deliberately made third
 > party modules easier to consume, thus weakening the convenience
 > argument that applies to stdlib inclusion.

Maybe.  That depends on if you care about the convenience of folks who
have to get new modules past Corporate Security, but it's easier to
get an upgrade of the whole shebang.  I don't think it's ever really
been resolved whether they're a "typical case that won't go away" or a
special group whose special needs should be considered.

Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 4): Python 2.7 network security enhancements

[Stephen J. Turnbull]

Alex Gaynor writes:

 > Here's my proposed list of such featuers:

And suppose that list grows over time?  After all, it once was [].

If we go for a feature-by-feature list, that has two more-or-less
hidden costs.  (1) Python-Dev has to specify which ones, and either
risks a new specification debate in the future, or needs to spend
time now describing criteria and processes for extending the list.
(2) Users may need to worry about the list.  (OTOH, as long as the
list is restricted to features in certain modules, users can choose to
assume anything in those modules may have changed behavior and that's
no different from Nick's proposal for them.)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com