Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Nikolaus Rath]

On Feb 25 2018, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Feb 25, 2018 at 11:02 PM, Nikolaus Rath <nikol...@rath.org> wrote:
>> On Feb 22 2018, Serhiy Storchaka <storch...@gmail.com> wrote:
>>> 1. Inner generator expression:
>>>
>>> result = [y + g(y) for y in (f(x) for x in range(10))]
>>>
>> [...]
>>>
>>> And maybe there are other ways.
>>
>> I think the syntax recently brough up by Nick is still the most
>> beautiful:
>>
>> result = [ (f(x) as y) + g(y) for x in range(10)]
>>
>> ..but I wonder if it is feasible to make the interpreter sufficiently
>> smart to evaluate the first summand before the second.
>
> It already has to. The order of evaluation in Python is well defined,
> mostly "left to right".

Ah, then the problem is how to evaluate

result = [ y + g(f(x) as y) for x in range(10)]

I don't think there'd be a good reason to allow one but not the other.

> But if you allow this in a comprehension, the
> obvious next step will be "do we allow this in ANY expression?"

Yes, of course. After all, IIRC Nick proposed it to simplify ternary
expressions.


Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Nikolaus Rath]

On Feb 22 2018, Serhiy Storchaka  wrote:
> 1. Inner generator expression:
>
> result = [y + g(y) for y in (f(x) for x in range(10))]
>
[...]
>
> And maybe there are other ways.

I think the syntax recently brough up by Nick is still the most
beautiful:

result = [ (f(x) as y) + g(y) for x in range(10)]

..but I wonder if it is feasible to make the interpreter sufficiently
smart to evaluate the first summand before the second.

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Nikolaus Rath]

On Jul 21 2017, David Mertz  wrote:
> How implausible is it to write out the actual memory image of a loaded
> Python process?

That is what Emacs does, and it causes them a lot of trouble. They're
trying to move away from it at the moment, but the direction is not yet
clear. The keyword is "unexec", and it wrecks havoc with malloc.

Best,
-Nikolaus
-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467

[Nikolaus Rath]

On Oct 11 2016, Nathaniel Smith  wrote:
> On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki  wrote:
>> From Python 3.4, bytearray is good solution for I/O buffer, thanks to
>> #19087 [1].
>> Actually, asyncio uses bytearray as I/O buffer often.
>
> Whoa what?! This is awesome, I had no idea that bytearray had O(1)
> deletes at the front. I literally reimplemented this myself on type of
> bytearray for some 3.5-only code recently because I assumed bytearray
> had the same asymptotics as list, and AFAICT this is totally
> undocumented. 

Indeed, same here.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drastically improving list.sort() for lists of strings/ints

[Nikolaus Rath]

On Sep 13 2016, Tim Peters <tim.pet...@gmail.com> wrote:
> [Terry Reedy <tjre...@udel.edu>]
>>> Tim Peters investigated and empirically determined that an
>>> O(n*n) binary insort, as he optimized it on real machines, is faster
>>> than O(n*logn) sorting for up to around 64 items.
>
> [Nikolaus Rath <nikol...@rath.org>]
>> Out of curiosity: is this test repeated periodically on different
>> architectures? Or could it be that it only ever was true 10 years ago on
>> Tim's Power Mac G5 (or whatever he used)?
>
> It has little to do with architecture, but much to do with the
> relative cost of comparisons versus pointer-copying.  Near the end of
>
> https://github.com/python/cpython/blob/master/Objects/listsort.txt
[...]

Ah, that makes sense, thanks! 

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Drastically improving list.sort() for lists of strings/ints

[Nikolaus Rath]

On Sep 11 2016, Terry Reedy  wrote:
> Tim Peters investigated and empirically determined that an
> O(n*n) binary insort, as he optimized it on real machines, is faster
> than O(n*logn) sorting for up to around 64 items.

Out of curiosity: is this test repeated periodically on different
architectures? Or could it be that it only ever was true 10 years ago on
Tim's Power Mac G5 (or whatever he used)?

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] When to use EOFError?

[Nikolaus Rath]

On Jun 21 2016, Serhiy Storchaka  wrote:
> There is a design question. If you read file in some format or with
> some protocol, and the data is ended unexpectedly, when to use general
> EOFError exception and when to use format/protocol specific exception?
>
> For example when load truncated pickle data, an unpickler can raise
> EOFError, UnpicklingError, ValueError or AttributeError. It is
> possible to avoid ValueError or AttributeError, but what exception
> should be raised instead, EOFError or UnpicklingError?

I think EOFError conveys more information. UnpicklingError can mean a
lot of things, EOFError tells you the precise problem: pickle expected
more data, but there was nothing left. I think in doubt the more
specific exception (in this case EOFError) should be raised.

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 16 2016, Nick Coghlan  wrote:
> On 16 June 2016 at 09:39, Paul Moore  wrote:
>> I'm willing to accept the view of the security experts that there's a
>> problem here. But without a clear explanation of the problem, how can
>> a non-specialist like myself have an opinion? (And I hope the security
>> POV isn't "you don't need an opinion, just do as we say").
>
> If you're not writing Linux (and presumably *BSD) scripts and
> applications that run during system initialisation or on embedded ARM
> hardware with no good sources of randomness, then there's zero chance
> of any change made in relation to this affecting you (Windows and Mac
> OS X are completely immune, since they don't allow Python scripts to
> run early enough in the boot sequence for there to ever be a problem).
>
> The only question at hand is what CPython should do in the case where
> the operating system *does* let Python scripts run before the system
> random number generator is ready, and the application calls a security
> sensitive API that relies on that RNG:
>
> - throw BlockingIOError (so the script developer knows they have a
> potential problem to fix)
> - block (so the script developer has a system hang to debug)
> - return low quality random data (so the script developer doesn't even
> know they have a potential problem)
>
> The last option is the status quo, and has a remarkable number of
> vocal defenders.

*applaud*


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Larry Hastings <la...@hastings.org> wrote:
> On 06/09/2016 07:38 PM, Nikolaus Rath wrote:
>> On Jun 09 2016, Larry Hastings <la...@hastings.org> wrote:
>>> Nope, I want the old behavior back.  os.urandom() should read
>>> /dev/random if getrandom() would block.  As the British say, "it
>>> should do what it says on the tin".
>> Aeh, what the tin says is "return random bytes".
>
> What the tin says is "urandom", which has local man pages that dictate
> exactly how it behaves. 
[...]

I disagree. The authoritative source for the behavior of the Python
'urandom' function is the Python documentation, not the Linux manpage
for the "urandom" device.

And https://docs.python.org/3.4/library/os.html says first and foremost:

 os.urandom(n)¶

Return a string of n random bytes suitable for cryptographic use.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Guido van Rossum  wrote:
> I don't think we should add a new function. I think we should convince
> ourselves that there is not enough of a risk of an exploit even if
> os.urandom() falls back.

That will be hard, because you have to consider an active, clever
adversary.

On the other hand, convincing yourself that in practice os.urandom would
never block unless the setup is super exotic or there is active
maliciousness seems much easier.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Nikolaus Rath]

On Jun 09 2016, Larry Hastings  wrote:
> On 06/09/2016 03:44 PM, Ethan Furman wrote:
>> On 06/09/2016 03:22 PM, Larry Hastings wrote:
>>> Okay, it's decided: os.urandom() must be changed for 3.5.2 to never
>>> block on a getrandom() call.
>>
>> One way to not block is to raise an exception.  Since this is such a
>> rare occurrence anyway I don't see this being a problem, plus it
>> keeps everybody mostly happy:  normal users won't see it hang,
>> crypto-folk won't see vulnerable-from-this-cause-by-default
>> machines, and those running Python early in the boot sequence will
>> have something they can figure out, plus an existing knob to work
>> around it [hashseed, I think?].
>
> Nope, I want the old behavior back.  os.urandom() should read
> /dev/random if getrandom() would block.  As the British say, "it
> should do what it says on the tin".

Aeh, what the tin says is "return random bytes". What everyone uses it
for (including the standard library) is to provide randomness for
cryptographic purposes. What it does (in the problematic case) is return
something that's not random.

To me this sounds about as sensible as having open('/dev/zero') return
non-zero values in some rare situations. And yes, for most people "the
kernel running out of zeros" makes exactly as much sense as "the kernel
runs out of random data". 


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] file system path protocol PEP

[Nikolaus Rath]

On May 11 2016, Brett Cannon  wrote:
> This PEP proposes a protocol for classes which represent a file system
> path to be able to provide a ``str`` or ``bytes`` representation.
[...]

As I said before, to me this seems like a lot of effort for a very
specific use-case. So let me put forward two hypothetical scenarios to
better understand your position:

- A new module for URL handling is added to the standard library (or
  urllib is suitably extended). There is a proposal to add a new
  protocol that allows classes to provide a ``str`` or ``bytes``
  representation of URLs.

- A new (third-party) library for natural language processing arises
  that exposes a specific class for representing audio data. Existing
  language processing code just uses bytes objects. To ease transition
  and interoperability, it is proposed to add a new protocol for classes
  that represend audio data to provide a bytes representation.

Do you think you would you be in favor of adding these protocols to
the stdlib/languange reference as well? If not, what's the crucial
difference to file system paths?


Thanks,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Nikolaus Rath]

On Apr 13 2016, Ethan Furman <et...@stoneleaf.us> wrote:
> On 04/13/2016 03:45 PM, Nikolaus Rath wrote:
>
>> When passing an object that is of type str and has a __fspath__
>> attribute, all approaches return the value of __fspath__().
>>
>> However, when passing something of type bytes, the second approach
>> returns the object, while the third returns the value of __fspath__().
>>
>> Is this intentional? I think a __fspath__ attribute should always be
>> preferred.
>
> Yes, it is intentional.  The second approach assumes __fspath__ can
> only contain str, so there is no point in checking it for bytes.

Either I haven't understood your answer, or you haven't understood my
question. I'm concerned about this case:

  class Special(bytes):
  def __fspath__(self):
return 'str-val'
  obj = Special('bytes-val', 'utf8')
  path_obj = fspath(obj, allow_bytes=True)  

With #2, path_obj == 'bytes-val'. With #3, path_obj == 'str-val'.

I would expect that fspath(obj, allow_bytes=True) == 'str-val' (after
all, it's allow_bytes, not require_bytes). Bu


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pathlib - current status of discussions

[Nikolaus Rath]

On Apr 13 2016, Brett Cannon  wrote:
> On Tue, 12 Apr 2016 at 22:38 Michael Mysinger via Python-Dev <
> python-dev@python.org> wrote:
>
>> Ethan Furman  stoneleaf.us> writes:
>>
>> > Do we allow bytes to be returned from os.fspath()?  If yes, then do we
>> > allow bytes from __fspath__()?
>>
>> De-lurking. Especially since the ultimate goal is better interoperability,
>> I
>> feel like an implementation that people can play with would help guide the
>> few remaining decisions. To help test the various options you could
>> temporarily add a _allow_bytes=GLOBAL_CONFIG_OPTION default argument to
>> both
>> pathlib.__fspath__() and os.fspath(), with distinct configurable defaults
>> for
>> each.
>>
>> In the spirit of Python 3 I feel like bytes might not be needed in
>> practice,
>> but something like this with defaults of False will allow people to easily
>> test all the various options.
>>
>
> https://gist.github.com/brettcannon/b3719f54715787d54a206bc011869aa1 has
> the four potential approaches implemented (although it doesn't follow the
> "separate functions" approach some are proposing and instead goes with the
> allow_bytes approach I originally proposed).


When passing an object that is of type str and has a __fspath__
attribute, all approaches return the value of __fspath__().

However, when passing something of type bytes, the second approach
returns the object, while the third returns the value of __fspath__().

Is this intentional? I think a __fspath__ attribute should always be
preferred.


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Nikolaus Rath]

On Apr 13 2016, Ethan Furman  wrote:
> (I'm not very good at keeping similar sounding functions separate --
> what's the difference between shutil.copy and shutil.copy2?  I have to
> look it up every time).

Well, "2" is more than "" (or 1), so copy2() copies *more* than copy() -
it includes the metadata. That always helps me.


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Nikolaus Rath]

On Apr 11 2016, Jon Ribbens  wrote:
>> What I see is that you asked to break your sandbox, and less than 1
>> hour later, a first vulnerability was found (exec called with two
>> parameters). A few hours later, a second vulnerability was found
>> (async generator and cr_frame).
>
> The former was just a stupid bug, it says nothing about the viability
> of the methodology. The latter was a new feature in a Python version
> later than I have ever used, and again does not imply anything much
> about the viability.

It implies that new versions of Python may break your sandbox. That
doesn't sound like a viable long-term solution.

> I think now I've blocked the names of frame
> object attributes it wouldn't be a vulnerability any more anyway.

It seems like you're playing whack-a-mole. 


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)

[Nikolaus Rath]

On Apr 10 2016, Jon Ribbens  wrote:
> On Sat, Apr 09, 2016 at 02:43:19PM +0200, Victor Stinner wrote:
>>Please don't loose time trying yet another sandbox inside CPython. It's
>>just a waste of time. It's broken by design.
>> 
>>Please read my email about my attempt (pysandbox):
>>https://lwn.net/Articles/574323/
>> 
>>And the LWN article:
>>https://lwn.net/Articles/574215/
>> 
>>There are a lot of safe ways to run CPython inside a sandbox (and not rhe
>>opposite).
>> 
>>I started as you, add more and more things to a blacklist, but it doesn't
>>work.
>
> That's the opposite of my approach though - I'm starting small and
> adding things, not starting with everything and removing stuff.

That contradicts what you said in another mail:


On Apr 08 2016, Jon Ribbens  wrote:
> Ah, I've not used Python 3.5, and I can't find any documentation on
> this cr_frame business, but I've added cr_frame and f_back to the
> disallowed attributes list.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Nikolaus Rath]

On Apr 07 2016, Donald Stufft <don...@stufft.io> wrote:
>> On Apr 7, 2016, at 6:48 AM, Nikolaus Rath <nikol...@rath.org> wrote:
>> 
>> Does anyone anticipate any classes other than those from pathlib to come
>> with such a method?
>
>
> It seems like it would be reasonable for pathlib.Path to call fspath on the
> path passed to pathlib.Path.__init__, which would mean that if other libraries
> implemented __fspath__ then you could pass their path objects to pathlib and
> it would just work (and similarly, if they also called fspath it would enable
> interoperation between all of the various path libraries).

Indeed, but my question is: is this actually going to happen? Are there
going to be other libraries that will implement __fspath__, and will
there be demand for pathlib to support them?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Defining a path protocol

[Nikolaus Rath]

On Apr 06 2016, Ethan Furman  wrote:
> On 04/06/2016 11:15 PM, Greg Ewing wrote:
>> Chris Barker - NOAA Federal wrote:
>>> But fspath(), if it exists, would call __fspath__ on an arbitrary
>>> object, and create a new string -- not a new Path. That may be
>>> confusing...
>>
>> Maybe something like fspathstr/__fspathstr__ would be better?
>
> As someone already said, we don't need to embed the type in the name.
>
> The point of the __os_path__ protocol is to return the serialized
> version of the Path the object represents.  This would be somewhat
> similar to the various __reduce*__ protocols (which I thought had
> something to do with adding until I learned what they were for).

Does anyone anticipate any classes other than those from pathlib to come
with such a method?

It seems odd to me to introduce a special method (and potentially a
buildin too) if it's only going to be used by a single module.

Why is:

path = getattr(obj, '__fspath__') if hasattr(obj, '__fspath__') else obj

better than

path = str(obj) if isinstance(obj, pathlib.Path) else obj

?

Yes, I know there are other pathlib-like modules out there. But isn't
pathlib meant to replace them?

Best,
Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Git for Mercurial Users

[Nikolaus Rath]

Hello,

With the upcoming move to Git, I thought people might be 
interested in some thoughts that I wrote down when learning Git 
for the first time as a long-time Mercurial user:


http://www.rath.org/mercurial-for-git-users-and-vice-versa.html

Comments are welcome (but probably more appropriate off-list).


Best,
-Nikolaus 


--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] More optimisation ideas

[Nikolaus Rath]

On Feb 01 2016, mike.romb...@comcast.net wrote:
" " == Barry Warsaw  writes: 


>> On Feb 01, 2016, at 11:40 AM, R. David Murray wrote: 
 
>> I don't know about anyone else, but on my own development 
>> systems it is not that unusual for me to *edit* the 
>> stdlib files (to add debug prints) while debugging my own 
>> programs.  Freeze would definitely interfere with that. 
>> I could, of course, install a separate source build on my 
>> dev system, but I thought it worth mentioning as a 
>> factor. 

   [snip] 

 > But even with system scripts, I do need to step through 
 > them occasionally.  If it were a matter of changing a 
 > shebang or invoking the script with a different Python 
 > (e.g. /usr/bin/python3s vs. /usr/bin/python3) to get the 
 > full unpacked source, that would be fine. 

  If the stdlib were to use implicit namespace packages 
( https://www.python.org/dev/peps/pep-0420/ ) and the various 
loaders/importers as well, then python could do what I've done 
with an embedded python application for years.  Freeze the 
stdlib (or put it in a zipfile or whatever is fast).  Then 
arrange PYTHONPATH to first look on the filesystem and then look 
in the frozen/ziped storage.


Presumably that would eliminate the performance advantages of the 
frozen/zipped storage because now Python would still have to issue 
all the stat calls to first check for the existence of a .py file.



Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Third milestone of FAT Python

[Nikolaus Rath]

On Dec 04 2015, Victor Stinner  wrote:
> Hi,
>
> I implemented 3 new optimizations in FAT Python: loop unrolling, constant
> folding and copy builtin functions to constants. In the previous thread,
> Terry Reedy asked me if the test suite is complete enough to ensure that
> FAT Python doesn't break the Python semantic.
[...]

I just wanted to say that I think this is truly great! Thanks working
on this!


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Paul Moore  wrote:
> On 16 September 2015 at 06:10, Stephen J. Turnbull  wrote:
>>   The only thing that hg really lost badly on
>> IMO was "named branches", and that's been fixed with bookmarks.
>
> FWIW, I still find bookmarks confusing to use compared to git
> branches. I don't know whether that's because bitbucket doesn't
> support them well, or whether I don't know all the details of
> bookmarks, but they just seem to get me in a muddle.
>
> For example, my usual workflow is
>
> (in a local clone of my fork on github)
>
> git checkout -b featureX
> ... hack ...
> git commit
> git push -u origin featureX # Push the local branch to github and set
> as remote tracking
>
> # later, on a  different PC
> git pull
> git checkout featureX # Sets up a remote tracking branch
> ... hack ...
> git commit
> git push
>
> # Finally, make a PR via the github UI
>
> # Once the PR is accepted
> git branch -d featureX # Remove my local branch, deleting all of the
> no longer required changesets
>
> I don't know of an equivalent of remote tracking branches in
> Mercurial. Maybe bookmarks work that way by default?

Where exactly did you run into problems? I think you should have gotten
the same result with the following hg commands (if your remote is
non-ancient):

.. hack ..
hg commit
hg bookmark featureX
hg push -B featureX origin

# later
hg pull -B featureX origin
... hack ..
hg commit
hg push


The "remote tracking branch" in Mercurial always exists, but it doesn't
have a special name. In hg, branches do not need to have names, they are
identified by their commit id. Assigning names is pure convenience, this
is why they're called "bookmarks".

> Also, my workflow involves 2 separate PCs, and I use my personal
> github forks to share work in progress between them. Mercurial makes
> it very difficult to edit "published" history, and I can't tell it
> that my bitbucket repo doesn't count as "published" (as far as I
> know).

In general you can do that by configuring the repository with

  [phases]
  publish = False

However, I believe BitBucket doesn't allow you to do that. But that's
not hg's fault.

> Git lets me do what I want (with some "are you sure?" prompts
> when I force-push a rebased/edited branch, for example).

Same with hg. "hg phase -d -f " forces the status of "rev" to
'draft'.


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Chris Angelico  wrote:
> With git, there are infinite workflows possible - you aren't forced to
> have a concept of "central server" and "clients" the way you would
> with SVN. Mercurial's called a DVCS too, so presumably it's possible
> to operate on a pure-peering model with no central server at all; how
> does that tie in with the inability to alter some commits?

There is no inability to do so in hg either, you just need some --force
flags ("hg phase -f -d " should be enough in almost all cases).


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, "R. David Murray"  wrote:
> The DAG plus git branches-as-labels *fits in my head* in a way that the
> DAG plus named-branches-and-other-things does not.

Hmm, that's odd. As far as I know, the difference between the hg and git
DAG model can be summarized like this:

 * In git, leaves of the DAG must be assigned a name. If they don't have
   a name, they will be garbage collected. If they have a name, they are
   called a branch.

 * In hg, leaves of the DAG persist. If you want to remove them, you
   have to do so explicitly (hg strip), if you want them to have a name,
   you must do so explicitly (hg bookmark). A node of the DAG with a
   name is called a bookmark.

 * hg named branches have no equivalent in git. 

Does that help?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Paul Moore <p.f.mo...@gmail.com> wrote:
> I'm on revision X, which is at tip (master in git, the default branch
> in Mercurial). I do git checkout -b foo (or hg bookmark foo). Now I
> check in some changes. In git, I'm on branch foo and I can go back to
> master with git checkout master. In Mercurial, I am at bookmark foo,
> but where's the place I started from? The main line of development
> (the default branch) doesn't have a bookmark, so I have to go hunting
> for the revision I first switched to the bookmark at.

Yes - but you could bookmark it before committing, then you don't have
to hunt for it :-).

$ hg pull
$ hg bookmark "my-origin-branch"
$ hg bookmark "my-local-branch"
$ hg update -r my-local-branch
$ hg commit
$ hg update -r my-origin-branch

> Similarly, if I'm at revision X, hg bookmark foo; hg bookmark bar
> creates 2 bookmarks. If I then check in a change, I guess *both*
> bookmarks move.

No, only the active bookmark moves automatically:

$ hg bookmark foo
$ hg bookmark bar
$ hg log -r tip
changeset:   0:d1c121e915b8
bookmark:bar
bookmark:foo
tag: tip
user:Nikolaus Rath <nikol...@rath.org>
date:Wed Sep 16 13:31:13 2015 -0700
files:   file
description:
initial commit


$ hg update -r foo
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
(activating bookmark foo)
$ echo barf > file
$ hg commit -m "Barf!"
file
committed changeset 1:ad44f9b935dc
$ hg log -r tip
changeset:   1:ad44f9b935dc
bookmark:foo
tag: tip
user:Nikolaus Rath <nikol...@rath.org>
date:Wed Sep 16 13:31:42 2015 -0700
files:   file
description:
Barf!

> In git, if I do git checkout -b foo; git checkout -b
> bar, I have created two branches, and checkins now only go on bar. The
> difference comes from the fact that in git, branches are "real
> things", but in hg, bookmarks are names for points on the DAG

You'll have to elaborate on that. What is it the difference between a
named leaf in the DAG and a branch that makes the branch "real"?

> See what I mean when I say I get confused? ;-)

I think what you mean is that you haven't read the Mercurial
documentation for quite some time :-).

>> The "remote tracking branch" in Mercurial always exists, but it doesn't
>> have a special name. In hg, branches do not need to have names, they are
>> identified by their commit id. Assigning names is pure convenience, this
>> is why they're called "bookmarks".
>
> And yet branches are *not* simply names in git - a branch with no
> commits on it is still distinct from its parent branch.
>
> So conceptually, the idea that hg bookmarks and git branches are the
> same thing, isn't actually true...

Well, the one thing were you thought they were different wasn't actually
the case. Is there another?

>>> Also, my workflow involves 2 separate PCs, and I use my personal
>>> github forks to share work in progress between them. Mercurial makes
>>> it very difficult to edit "published" history, and I can't tell it
>>> that my bitbucket repo doesn't count as "published" (as far as I
>>> know).
>>
>> In general you can do that by configuring the repository with
>>
>>   [phases]
>>   publish = False
>>
>> However, I believe BitBucket doesn't allow you to do that. But that's
>> not hg's fault.
>
> Well, yes and no. The fact that you have to edit the remote repository
> before it will allow you to violate its "history is immutable"

That's not true. You have to edit the remote repository only if you want
to edit history without --force.


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 17 2015, "Stephen J. Turnbull" <step...@xemacs.org> wrote:
> Nikolaus Rath writes:
>
>  > Hmm, that's odd. As far as I know, the difference between the hg and git
>  > DAG model can be summarized like this:
>  > 
>  >  * In git, leaves of the DAG must be assigned a name. If they don't have
>  >a name, they will be garbage collected.
>
> You can turn off automatic garbage collection.  I usually do: it's
> very unusual that I create millions of objects, or even megabytes
> worth of objects, that I'm creating.

Okay... (I don't quite see why this matters here).


>  >If they have a name, they are called a branch.
>
> Tags are also refs, the difference being that committing child of the
> tip of the current branch advances the branch pointer, while that
> won't happen with a tag.

Yeah, it's like that both in hg and git, so I'm not quite sure what
you're trying to say...


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, "R. David Murray" <rdmur...@bitdance.com> wrote:
> On Wed, 16 Sep 2015 09:17:38 -0700, Nikolaus Rath <nikol...@rath.org> wrote:
>> On Sep 16 2015, "R. David Murray" <rdmur...@bitdance.com> wrote:
>> > The DAG plus git branches-as-labels *fits in my head* in a way that the
>> > DAG plus named-branches-and-other-things does not.
>> 
>> Hmm, that's odd. As far as I know, the difference between the hg and git
>> DAG model can be summarized like this:
>> 
>>  * In git, leaves of the DAG must be assigned a name. If they don't have
>>a name, they will be garbage collected. If they have a name, they are
>>called a branch.
>> 
>>  * In hg, leaves of the DAG persist. If you want to remove them, you
>>have to do so explicitly (hg strip), if you want them to have a name,
>>you must do so explicitly (hg bookmark). A node of the DAG with a
>>name is called a bookmark.
>> 
>>  * hg named branches have no equivalent in git. 
>> 
>> Does that help?
>
> Well, that last bullet kind of complicates the model, doesn't it?  :)

Not if you come from git and want to use hg. You can just ignore
bookmarks.

But there is an easy explanation if you want one. Think of a named
branch as automatically appending "this went to " to your
commit message.


> Especially when you add the fact that (IIUC) which named branch a commit
> is on is recorded in the commit or something, which means the DAG is
> more complicated than just being a DAG of commits

No, it's just some extra information in the commit. Like author, date,
or commit message.

> The fact that I have to worry about (remember to delete) branches I no
> longer want is also an additional mental load,

Yes, this is a disadvantage of Mercurial.

> especially since
> (again, IIUC) I'd have to figure out which commit I wanted to strip
> *from* in order to get rid of an abandoned branch.

..but this is not. You can easily automate that (though I don't
recommend typing in the expression by hand every time, the idea is to
tel hg to "delete this changeset and all ancestors until you reach an
ancestor that has more than one child").

> This is what I mean by hg not being *oriented* toward the simple model:
> if I end up with extra heads in my cpython repo I treat this as a bug
> that needs to be fixed.

But why?

> In git, it's just a branch I'm working with and
> later do something with...or delete, and git takes care of cleaning up
> the orphaned commits for me.

In hg it's exactly the same, it's just a head you're working with, and
later you can do something with it.. or delete it. If it helps, create a
cronjob that deletes all DAG leaves that don't have bookmarks.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Terry Reedy  wrote:
> On 9/16/2015 5:20 AM, Oleg Broytman wrote:
>
>> On Tue, Sep 15, 2015 at 07:44:28PM +, Augie Fackler  
>> wrote:
>
>>> There are a lot of reasons to prefer one tool over another. Common ones are
>>> familiarity, simplicity, and power.
>>
>> Add here documentation, speed, availability of extensions and
>> 3rd-party tools, hosting options (both locally installable and web
>> services).
>
> For me, the killer 3rd party tool in favor of hg is TortoiseHg, which
> I use on Windows. As far as I know (I did check a bit), there is no
> equivalent for git on Windows.

.. or elsewhere. Tortoisehg rocks.

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Nikolaus Rath <nikol...@rath.org> wrote:
> On Sep 16 2015, "R. David Murray" <rdmur...@bitdance.com> wrote:
>> On Wed, 16 Sep 2015 09:17:38 -0700, Nikolaus Rath <nikol...@rath.org> wrote:
>>> On Sep 16 2015, "R. David Murray" <rdmur...@bitdance.com> wrote:
>>> > The DAG plus git branches-as-labels *fits in my head* in a way that the
>>> > DAG plus named-branches-and-other-things does not.
>>> 
>>> Hmm, that's odd. As far as I know, the difference between the hg and git
>>> DAG model can be summarized like this:
>>> 
>>>  * In git, leaves of the DAG must be assigned a name. If they don't have
>>>a name, they will be garbage collected. If they have a name, they are
>>>called a branch.
>>> 
>>>  * In hg, leaves of the DAG persist. If you want to remove them, you
>>>have to do so explicitly (hg strip), if you want them to have a name,
>>>you must do so explicitly (hg bookmark). A node of the DAG with a
>>>name is called a bookmark.
>>> 
>>>  * hg named branches have no equivalent in git. 
>>> 
>>> Does that help?
>>
>> Well, that last bullet kind of complicates the model, doesn't it?  :)
>
> Not if you come from git and want to use hg. You can just ignore
> bookmarks.

Obviously, that should have read "..you can just ignore named branches".


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP: Collecting information about git

[Nikolaus Rath]

On Sep 16 2015, Chris Angelico  wrote:
> On Wed, Sep 16, 2015 at 5:44 AM, Augie Fackler  wrote:
>>> but git is still better at it: ``git add -p``
>>> allows me to review and edit patches before commit while ``hg record``
>>> commits immediately.
>>
>> FWIW, I totally *get* wanting a staging area. That said, other than the
>> staging area, record (aka commit --interactive) and git add -p are identical
>> functionality-wise. We also now ship (at least as of 3.5) a curses UI for
>> record, which is quite nice.
>
> Looks like it's time I spun up my own hg, rather than using the 3.1.2
> that ships with Debian. A better UI for interactive (partial) commits
> would go a long way toward filling the hole left by not having a
> staging area;

You don't need to update Mercurial, it sufficient to install the
Tortoisehg frontend (unless you really want a curses interface,
Tortoisehg is X11). Give it a shot!

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 501 Shell Command Examples

[Nikolaus Rath]

On Sep 05 2015, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 5 September 2015 at 12:36, Nikolaus Rath <nikol...@rath.org> wrote:
>> Hi Nick,
>>
>> You are giving
>>
>>   runcommand(sh(i"cat {filename}"))
>>
>> as an example that avoids injection attacks. While this is true, I think
>> this is still a terrible anti-pattern[1] that should not be entombed in
>> a PEP as a positive example.
>>
>> Could you consider removing it?
>>
>> (It doubly wastes resources by pointlessly calling a shell, and then by
>> parsing & quoting the argument only for the shell to do the same in
>> reverse).
>
> Any reasonable implementation of that pattern wouldn't actually call a
> system shell, it would invoke something like Julia's command system.

That's obvious to someone like you who thinks about this in terms of the
best implementation.

To someone less experienced, or just coming at from a different angle,
this example just says "writing a shell command is a good way to start
an external program, as long as I take care of quoting".


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 498: Literal String Interpolation is ready for pronouncement

[Nikolaus Rath]

On Sep 04 2015, "Eric V. Smith"  wrote:
> I've made a number of small changes to PEP 498. I don't think any of the
> changes I've made in the last week are substantive. Mostly I've
> clarified how it works and removing some limitations. The only
> meaningful change is that expressions are now surrounded by parens
> before they're evaluated. This allows for newlines inside of the
> expressions to be ignored.
>
> I think it's now ready for Guido to pronounce on it.

I just wanted to say that I think this is fantastic work. If the PEP
gets accepted, I expect that in my code f-strings will replace the other
interpolation mechanism in almost all cases.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 501 Shell Command Examples

[Nikolaus Rath]

Hi Nick,

You are giving

  runcommand(sh(i"cat {filename}"))

as an example that avoids injection attacks. While this is true, I think
this is still a terrible anti-pattern[1] that should not be entombed in
a PEP as a positive example.

Could you consider removing it?

(It doubly wastes resources by pointlessly calling a shell, and then by
parsing & quoting the argument only for the shell to do the same in
reverse).

Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-498: Literal String Formatting

[Nikolaus Rath]

On Aug 16 2015, Paul Moore p.f.mo...@gmail.com wrote:
 2. By far and away the most common use for me would be things like
 print(fIteration {n}: Took {end-start) seconds).

I believe an even more common use willl be

print(fIteration {n+1}: Took {end-start} seconds)

Note that not allowing expressions would turn this into the rather
verbose:

iteration=n+1
duration=end-start
print(fIteration {iteration}: Took {duration} seconds)


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-498: Literal String Formatting

[Nikolaus Rath]

On Aug 08 2015, Nick Coghlan ncogh...@gmail.com wrote:
 On 8 August 2015 at 11:39, Eric V. Smith e...@trueblade.com wrote:
 Following a long discussion on python-ideas, I've posted my draft of
 PEP-498. It describes the f-string approach that was the subject of
 the Briefer string format thread. I'm open to a better title than
 Literal String Formatting.

 Thanks you for your work on this - it's a very cool concept!

 I've also now written and posted an initial draft of PEP 500,
[...]

I think what that PEP really needs is a concise summary of the
*differences* to PEP 498. 

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-498: Literal String Formatting

[Nikolaus Rath]

On Aug 08 2015, Nikolaus Rath nikol...@rath.org wrote:
 On Aug 08 2015, Nick Coghlan ncogh...@gmail.com wrote:
 On 8 August 2015 at 11:39, Eric V. Smith e...@trueblade.com wrote:
 Following a long discussion on python-ideas, I've posted my draft of
 PEP-498. It describes the f-string approach that was the subject of
 the Briefer string format thread. I'm open to a better title than
 Literal String Formatting.

 Thanks you for your work on this - it's a very cool concept!

 I've also now written and posted an initial draft of PEP 500,
 [...]

 I think what that PEP really needs is a concise summary of the
 *differences* to PEP 498.

I should probably elaborate on that.

After reading both PEPs, it seems to me that the only difference is that
you want to use a different prefix (i instead of f), use ${} instead of
{}, and call a builtin function to perform the interpolation (instead of
always using format). But is that really it? The PEP appears rather
long, so I'm not sure if I'm missing other differences in the parts that
seemed identical to PEP 498 to me.


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 documentation

[Nikolaus Rath]

Hi,

No, not a browser cache issue. I was looking for async or await in the
table of contents, so I didn't notice the new coroutines sections.

Sorry for the noise.
-Nikolaus

On Aug 02 2015, Yury Selivanov yselivanov...@gmail.com wrote:
 Nikolaus,

 Strange.  PEP 492 changes are fully documented since b3.

 Here are just few examples:

 https://docs.python.org/3.5/whatsnew/3.5.html#pep-492-coroutines-with-async-and-await-syntax
 https://docs.python.org/3.5/reference/datamodel.html#coroutines
 https://docs.python.org/3.5/reference/compound_stmts.html#coroutines

 Perhaps, it's a browser cache issue?

 Yury

 On 2015-08-02 12:38 AM, Nikolaus Rath wrote:
 Hello,

 Looking at the language reference for 3.5.0b4, I noticed that it
 mentions neither async nor await.

 Is this still going to get updated, or will the only documentation
 consist of the PEP itself? I think having a Python release recognize
 keywords that are not mentioned in the language reference would be quite
 unfortunate (even if they're treated specially to preserve backwards
 compatibility).

 Best,
 -Nikolaus




-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 492 documentation

[Nikolaus Rath]

Hello,

Looking at the language reference for 3.5.0b4, I noticed that it
mentions neither async nor await.

Is this still going to get updated, or will the only documentation
consist of the PEP itself? I think having a Python release recognize
keywords that are not mentioned in the language reference would be quite
unfortunate (even if they're treated specially to preserve backwards
compatibility).

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issues not responded to.

[Nikolaus Rath]

On Jul 31 2015, Mark Lawrence breamore...@yahoo.co.uk wrote:
 There are over 400 issues on the bug tracker that have not had a
 response to the initial message, roughly half of these within the last
 eight months alone.  Is there a (relatively) simple way that we can
 share these out between us to sort those that are likely to need
 dealing with in the medium to longer term, from the simple short term
 ones, e.g very easy typo fixes?


Nick recently mentioned that the PSF might be able to help, but that the
initiative for that needs to come from the core developers. So why don't
you guys ask the PSF to e.g. sponsor some of the work that no one feels
motivated to do in their spare time?

To avoid issues with some people being paid for work that others
contribute in their free time one could introduce a new keyword in the
tracker (say ugly). Whenever a core developer sees an issue that he[1]
thinks should be worked on, but that he really does not want to do in
his free time, he tags it with ugly and the issue becomes available
for PSF-sponsored work.

Best,
-Nikolaus

[1] I first wanted to write he/she - but are there actually any female
core contributors?

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Nikolaus Rath]

On Jul 27 2015, Lennart Regebro rege...@gmail.com wrote:
 That you add one hour to it, and the datetime moves forward one hour
 in actual time? That's doable, but during certain circumstance this
 may mean that you go from 1AM to 1AM, or from 1AM to 3AM.

 Or do you expect that adding one hour will increase the hour count
 with one, ie that the wall time increases with one hour? This may
 actually leave you with a datetime that does not exist, so that is not
 something you can consistently do.

Apologies for asking yet another dumb question about this, but I have
the impression that a lot of other people are struggling with the basics
here too.

Can you tell us which of the two operations datetime currently
implements?

And when people talk about explicitly converting to UTC and back, does
that mean that if you're (again, with the current implementation)
converting to UTC, *then* add the one hour, and then convert back, you
get the other operation (that you don't get when you directly add 1
day)?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How do we tell if we're helping or hindering the core development process?

[Nikolaus Rath]

On Jul 22 2015, Nick Coghlan ncogh...@gmail.com wrote:
 On 22 July 2015 at 13:23, Nikolaus Rath nikol...@rath.org wrote:
 If it were up to me, I'd focus all the resources of the PSF on reducing
 this backlog - be that by hiring some core developers to work full-time
 on just the open bugtracker issues, or by financing development of
 better code review and commit infrastructure.

 Ah, but the PSF can't do that without infringing on python-dev's
 autonomy - switching to my PSF Director's hat, while we'd certainly be
 prepared to help with funding a credible grant proposal for something
 like the Twisted technical fellowship, we wouldn't *impose* help that
 the core developers haven't asked for.

I don't understand. If I would hire a core developer myself to work on
this (theoretically, I have no plans to do that), would that also be
infringing python-dev's authority? If so, how is that different from me
doing the work? If not, why is it different if the PSF decides to hire
someone?

 The current situation looks like a downward spiral to me. New
 contributors are frustrated and leave because they feel their
 contribution is not welcome, and core developers get burned out by
 the gigantic backlog and the interaction with frustrated patch
 submitters - thus further reducing the available manpower.

 We actually still have a lot of paid core developer (and potential
 core developer) time locked up in facilitating the Python 2 - 3
 migration, as we didn't fully appreciate the extent to which Python
 had been adopted in the Linux ecosystem and elsewhere until folks
 started seeking help upgrading.

Interesting. Is this information available publically somewhere? I'm
curious what exactly is being worked on.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How do we tell if we're helping or hindering the core development process?

[Nikolaus Rath]

On Jul 21 2015, Nick Coghlan ncogh...@gmail.com wrote:
 All of this is why the chart that I believe should be worrying people
 is the topmost one on this page:
 http://bugs.python.org/issue?@template=stats

 Both the number of open issues and the number of open issues with
 patches are steadily trending upwards. That means the bottleneck in
 the current process *isn't* getting patches written in the first
 place, it's getting them up to the appropriate standards and applied.
 Yet the answer to the problem isn't a simple recruit more core
 developers, as the existing core developers are *also* the bottleneck
 in the review and mentoring process for *new* core developers.

As a random datapoint:

Early last year I wanted get involved in CPython development. In the
next months I submitted and reviewed maybe 20 - 25 patches in the bug
tracker. After seeing all of them languish, I stopped submitting and
reviewing and just tried to get someone to look at the issues that I'd
worked on. Eventually, I managed to get almost all of them committed
(the last one sometime this February, after more than a year). However,
it was such an uphill battle just to get someone to look at my
contributions that I haven't touched the bugtracker ever since.

If it were up to me, I'd focus all the resources of the PSF on reducing
this backlog - be that by hiring some core developers to work full-time
on just the open bugtracker issues, or by financing development of
better code review and commit infrastructure. The current situation
looks like a downward spiral to me. New contributors are frustrated and
leave because they feel their contribution is not welcome, and core
developers get burned out by the gigantic backlog and the interaction
with frustrated patch submitters - thus further reducing the available
manpower.

(I am aware of the forge PEPs that you mention below, but as far as I
know even those are stalled because of - guess what - lack of available
core developer time).

Working on Roundup, the CPython infrastructure or Kallithea may be less
frustrating for new contributors (I don't know), but what this boils
down to is that you're contributing to a different project, and not to
CPython. But if you've decided to work on something else, there is (at
least for me) little reason to restrict the choice to projects that are
used *for* CPython development. And compared to the whole range of other
open source projects, I suspect the above options just don't look all
that interesting to many people. In other words: you typically can't
tell volunteers what to work on - you've got to accept the contribution
in the area they're interested in, or you loose the contributor. In case
of CPython the latter may be unavoidable at the moment, but I think it's
illusory to think that this can be solved by redirecting the stream of
contributions. Suppose you just found a bug in Python and you want to
upstream the patch so you don't have to carry the workaround
forever. Now you see there are already a lot of open issues with patches
- are you going to forget about your patch and e.g. decide to work on
the buildbots insteads?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Nikolaus Rath]

On Apr 20 2015, Chris Angelico ros...@gmail.com wrote:
 Maybe it'd be of value to have a quick code stripper that takes away
 all the annotations, plus any other junk/framing that you're not
 interested in, and gives you something you can browse in a text
 editor?

If you need to preprocess your source code to make it suitable for human
consumption something is very wrong with your language design. I can't
believe you're seriously suggesting this.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Nikolaus Rath]

On Apr 21 2015, Paul Sokolovsky pmis...@gmail.com wrote:
 Hello,

 On Tue, 21 Apr 2015 08:05:59 -0700
 Nikolaus Rath nikol...@rath.org wrote:

 On Apr 20 2015, Chris Angelico ros...@gmail.com wrote:
  Maybe it'd be of value to have a quick code stripper that takes
  away all the annotations, plus any other junk/framing that you're
  not interested in, and gives you something you can browse in a text
  editor?
 
 If you need to preprocess your source code to make it suitable for
 human consumption something is very wrong with your language design.
 I can't believe you're seriously suggesting this.

 I'm sure that was irony, d'oh.

That'd be a relief. It didn't sound ironic to me.

 The proposed type annotations are very readable.
[..]

I don't have an informed opinion about that yet. I was just commenting
on the general idea of stripping them away if they're not readable.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type hints -- a mediocre programmer's reaction

[Nikolaus Rath]

On Apr 20 2015, Harry Percival hj...@cantab.net wrote:
 My first reaction to type hints was yuck, and I'm sure I'm not the only
 one to think that.  viz (from some pycon slides):

 def zipmap(f: Callable[[int, int], int], xx: List[int],
yy: List[int]) - List[Tuple[int, int, int]]:

 arg.  and imagine it with default arguments.

This is indeed ugly as hell and I certainly would not want to see this
in any Python file I'm working with.

However, I always assumed that whatever tools consume these annotations
are expected to be good enough at automatic type inference that
something like this would never be necessary?

In practice, I would hope that the above simplifies to

def zipmap(f, xx: List[int], yy: List[int]):
   
because (just picking a probably buggy random implementation)

   zz = [] # -- z must be List[A]
   for i in range(min(len(xx), len(yy))):
  x = xx[i]   # -- x must be int
  y = xx[i]   # -- y must be int
  z = f(x,y)  # -- f must be Callable[(int,int], B]
  zz[i] = (x,y,z)   # -- A must be Tuple[int,int,B]

return zz # -- return value must be List[Tuple[int,int,B]]

it doesn't catch that B = int, but I think that's acceptable.


Is this not the case? Are we really expecting people to write stuff like
the above?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Nikolaus Rath]

Nick Coghlan ncogh...@gmail.com writes:
 As some examples of where bilingual computing breaks down:

 * My NFS client and server may have different locale settings
 * My FTP client and server may have different locale settings
 * My SSH client and server may have different locale settings
 * I save a file locally and send it to someone with a different locale
 setting
 * I attempt to access a Windows share from a Linux client (or
 vice-versa)
 * I clone my POSIX hosted git or Mercurial repository on a Windows
 client
 * I have to connect my Linux client to a Windows Active Directory
 domain (or vice-versa)
 * I have to interoperate between native code and JVM code

 The entire computing industry is currently struggling with this
 monolingual (ASCII/Extended ASCII/EBCDIC/etc) - bilingual (locale
 encoding/code pages) - multilingual (Unicode) transition. It's been
 going on for decades, and it's still going to be quite some time
 before we're done.

 The POSIX world is slowly clawing its way towards a multilingual model
 that actually works: UTF-8
 Windows (including the CLR) and the JVM adopted a different
 multilingual model, but still one that actually works: UTF-16-LE


 Nick, I think the first half of your post is one of the clearest
 expositions yet of 'why Python 3' (in particular, the str to unicode
 change).  It is worthy of wider distribution and without much change, it
 would be a great blog post.

 Indeed, I had the same idea - I had been assuming users already understood
 this context, which is almost certainly an invalid assumption.

 The blog post version is already mostly written, but I ran out of weekend.
 Will hopefully finish it up and post it some time in the next few days
 :)

In that case, maybe it'd be nice to also explain why you use the term
bilingual for codepage based encoding. At least to me, a
codepage/locale is pretty monolingual, or alternatively covering a whole
region (e.g. western europe). I figure with bilingual you mean ascii +
something, but that's mostly a guess from my side.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Nikolaus Rath]

Chris Barker chris.bar...@noaa.gov writes:
 What I fail to see is why it's better to raise an exception and point users
 to a better way, than to simply provide an optimization so that it's a mute
 issue.

 The only justification offered here is that will teach people that summing
 strings (and some other objects?) is order(N^2) and a bad idea. But:

 a) Python's primary purpose is practical, not pedagogical (not that it
 isn't great for that)

 b) I doubt any naive users learn anything other than I can't use sum() for
 strings, I should use .join(). Will they make the leap to I shouldn't
 use string concatenation in a loop, either? Oh, wait, you can use string
 concatenation in a loop -- that's been optimized. So will they learn: some
 types of object shave poor performance with repeated concatenation and
 shouldn't be used with sum(). So If I write such a class, and want to sum
 them up, I'll need to write an optimized version of that code?

 I submit that no naive user is going to get any closer to a proper
 understanding of algorithmic Order behavior from this small hint. Which
 leaves no reason to prefer an Exception to an optimization.

 One other point: perhaps this will lead a naive user into thinking --
 sum() raises an exception if I try to use it inefficiently, so it must be
 OK to use for anything that doesn't raise an exception -- that would be a
 bad lesson to mis-learn

AOL to that.

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Commit-ready patches in need of review

[Nikolaus Rath]

Hello,

The following commit-ready patches have been waiting for review since
May and earlier.It'd be great if someone could find the time to take a
look. I'll be happy to incorporate feedback as necessary:

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)

* http://bugs.python.org/issue15955 (gzip, bz2, lzma: add option to
  limit output size)

* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

  I only wrote the patch for one file because I'd like to have feedback
  before tackling the second. However, the patches are independent so
  unless there are other problems this is ready for commit.


Best,
Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[Nikolaus Rath]

Ben Hoyt benh...@gmail.com writes:
 So here's the ways in which option #2 is now more complicated than option #1:

 1) it has an additional info argument, the values of which have to
 be documented ('os', 'type', 'lstat', and what each one means)
 2) it has an additional onerror argument, the signature of which and
 fairly complicated return value is non-obvious and has to be
 documented
 3) it requires user modification of the DirEntry object, which needs
 documentation, and is potentially hard to implement
 4) because the DirEntry object now allows modification, you need a
 stat_result() helper function to help you build your own stat values

 I'm afraid points 3 and 4 here add way too much complexity.

Points 3 and 4 are not required to go with option #2, option #2 merely
allows to implement points 3 and 4 at some point in the future if it
turns out to be desirable.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does _pyio.*.readinto have to work with 'b' arrays?

[Nikolaus Rath]

Nick Coghlan ncogh...@gmail.com writes:
 On 15 June 2014 14:57, Nikolaus Rath nikol...@rath.org wrote:
 On 06/14/2014 09:31 PM, Nick Coghlan wrote:
 On 15 June 2014 10:41, Benjamin Peterson benja...@python.org wrote:
 On Sat, Jun 14, 2014, at 15:39, Nikolaus Rath wrote:
 It seems to me that a much cleaner solution would be to simply declare
 _pyio's readinto to only work with bytearrays, and to explicitly raise a
 (more helpful) TypeError if anything else is passed in.

 That seems reasonable. I don't think _pyio's behavior is terribly
 important compared to the C _io module.

 _pyio was written before the various memoryview fixes that were
 implemented in Python 3.3 - it seems to me it would make more sense to
 use memoryview to correctly handle arbitrary buffer exporters (we
 implemented similar fixes for the base64 module in 3.4).

 Definitely. But is there a way to do that without writing C code?

 Yes, Python level reshaping and typecasting of memory views is one of
 the key enhancements Stefan implemented for 3.3.
[..]

Ah, nice. I'll use that. Thank you Stefan :-).


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does _pyio.*.readinto have to work with 'b' arrays?

[Nikolaus Rath]

Victor Stinner victor.stin...@gmail.com writes:
 Le 15 juin 2014 02:42, Benjamin Peterson benja...@python.org a écrit :
 On Sat, Jun 14, 2014, at 15:39, Nikolaus Rath wrote:
  It seems to me that a much cleaner solution would be to simply declare
  _pyio's readinto to only work with bytearrays, and to explicitly raise a
  (more helpful) TypeError if anything else is passed in.

 That seems reasonable. I don't think _pyio's behavior is terribly
 important compared to the C _io module.

 Which types are accepted by the readinto() method of the C io module?

Everything implementing the buffer protocol.

 If the C module only accepts bytearray, the array hack must be removed
 from _pyio.

_pyio currently accepts only bytearray and 'b'-type arrays. But it seems
with memoryview.cast() we now have a way to make it behave like the C
module.


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Why does _pyio.*.readinto have to work with 'b' arrays?

[Nikolaus Rath]

Hello,

The _pyio.BufferedIOBase class contains the following hack to make sure
that you can read-into array objects with format 'b':

try:
b[:n] = data
except TypeError as err:
import array
if not isinstance(b, array.array):
raise err
b[:n] = array.array('b', data)

I am now wondering if I should implement the same hack in BufferedReader
(cf. issue 20578). Is there anything special about 'b' arrays that
justifies to treat them this way?

Note that readinto is supposed to work with any object implementing the
buffer protocol, but the Python implementation only works with
bytearrays and (with the above hack) 'b' arrays. Even using a 'B' array
fails:

 import _pyio
 from array import array
 buf = array('b', b'x' * 10)
 _pyio.open('/dev/zero', 'rb').readinto(buf) 
10
 buf = array('B', b'x' * 10)
 _pyio.open('/dev/zero', 'rb').readinto(buf)
Traceback (most recent call last):
  File /home/nikratio/clones/cpython/Lib/_pyio.py, line 662, in readinto
b[:n] = data
TypeError: can only assign array (not bytes) to array slice

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File stdin, line 1, in module
  File /home/nikratio/clones/cpython/Lib/_pyio.py, line 667, in readinto
b[:n] = array.array('b', data)
TypeError: bad argument type for built-in operation


It seems to me that a much cleaner solution would be to simply declare
_pyio's readinto to only work with bytearrays, and to explicitly raise a
(more helpful) TypeError if anything else is passed in.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does _pyio.*.readinto have to work with 'b' arrays?

[Nikolaus Rath]

On 06/14/2014 09:31 PM, Nick Coghlan wrote:
 On 15 June 2014 10:41, Benjamin Peterson benja...@python.org wrote:
 On Sat, Jun 14, 2014, at 15:39, Nikolaus Rath wrote:
 It seems to me that a much cleaner solution would be to simply declare
 _pyio's readinto to only work with bytearrays, and to explicitly raise a
 (more helpful) TypeError if anything else is passed in.

 That seems reasonable. I don't think _pyio's behavior is terribly
 important compared to the C _io module.
 
 _pyio was written before the various memoryview fixes that were
 implemented in Python 3.3 - it seems to me it would make more sense to
 use memoryview to correctly handle arbitrary buffer exporters (we
 implemented similar fixes for the base64 module in 3.4).

Definitely. But is there a way to do that without writing C code?

My attempts failed:

 from array import array
 a = array('b', b'x'*10)
 am = memoryview(a)
 am[:3] = b'foo'
Traceback (most recent call last):
  File stdin, line 1, in module
ValueError: memoryview assignment: lvalue and rvalue have different
structures
 am[:3] = memoryview(b'foo')
Traceback (most recent call last):
  File stdin, line 1, in module
ValueError: memoryview assignment: lvalue and rvalue have different
structures
 am.format = 'B'
Traceback (most recent call last):
  File stdin, line 1, in module
AttributeError: attribute 'format' of 'memoryview' objects is not writable

The only thing that works is:

 am[:3] = array('b', b'foo')

but that's again specific to a being a 'b'-array.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does IOBase.__del__ call .close?

[Nikolaus Rath]

Benjamin Peterson benja...@python.org writes:
 On Thu, Jun 12, 2014, at 18:06, Nikolaus Rath wrote:
 Consider this simple example:
 
 $ cat test.py 
 import io
 import warnings
 
 class StridedStream(io.IOBase):
 def __init__(self, name, stride=2):
 super().__init__()
 self.fh = open(name, 'rb')
 self.stride = stride
 
 def read(self, len_):
 return self.fh.read(self.stride*len_)[::self.stride]
 
 def close(self):
 self.fh.close()
 
 class FixedStridedStream(StridedStream):
 def __del__(self):
 # Prevent IOBase.__del__ frombeing called.
 pass
 
 warnings.resetwarnings()
 warnings.simplefilter('error')
 
 print('Creating  loosing StridedStream..')
 r = StridedStream('/dev/zero')
 del r
 
 print('Creating  loosing FixedStridedStream..')
 r = FixedStridedStream('/dev/zero')
 del r
 
 $ python3 test.py 
 Creating  loosing StridedStream..
 Creating  loosing FixedStridedStream..
 Exception ignored in: _io.FileIO name='/dev/zero' mode='rb'
 ResourceWarning: unclosed file _io.BufferedReader name='/dev/zero'
 
 In the first case, the destructor inherited from IOBase actually
 prevents the ResourceWarning from being emitted.

 Ah, I see. I don't see any good ways to fix it, though, besides setting
 some flag if close() is called from __del__.

How about not having IOBase.__del__ call self.close()? Any resources
acquired by the derived class would still clean up after themselves when
they are garbage collected.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does IOBase.__del__ call .close?

[Nikolaus Rath]

Benjamin Peterson benja...@python.org writes:
 On Wed, Jun 11, 2014, at 17:11, Nikolaus Rath wrote:
 MRAB pyt...@mrabarnett.plus.com writes:
  On 2014-06-11 02:30, Nikolaus Rath wrote:
  Hello,
 
  I recently noticed (after some rather protacted debugging) that the
  io.IOBase class comes with a destructor that calls self.close():
 
  [0] nikratio@vostro:~/tmp$ cat test.py
  import io
  class Foo(io.IOBase):
   def close(self):
   print('close called')
  r = Foo()
  del r
  [0] nikratio@vostro:~/tmp$ python3 test.py
  close called
 
  To me, this came as quite a surprise, and the best documentation of
  this feature seems to be the following note (from the io library
  reference):
 
  The abstract base classes also provide default implementations of some
methods in order to help implementation of concrete stream classes. For
example, BufferedIOBase provides unoptimized implementations of
readinto() and readline().
 
  For me, having __del__ call close() does not qualify as a reasonable
  default implementation unless close() is required to be idempotent
  (which one could deduce from the documentation if one tries to, but it's
  far from clear).
 
  Is this behavior an accident, or was that a deliberate decision?
 
  To me, it makes sense. You want to make sure that it's closed, releasing
  any resources it might be holding, even if you haven't done so
  explicitly.
 
 I agree with your intentions, but I come to the opposite conclusion:
 automatically calling close() in the destructor will hide that there's a
 problem in the code. Without that automatic cleanup, there's at least a
 good chance that a ResourceWarning will be emitted so the problem gets
 noticed. Silently work around bugs in caller's code doesn't seem like
 a very useful default to me...

 Things which actually hold system resources (like FileIO) give
 ResourceWarning if they close in __del__, so I don't understand your
 point.

Consider this simple example:

$ cat test.py 
import io
import warnings

class StridedStream(io.IOBase):
def __init__(self, name, stride=2):
super().__init__()
self.fh = open(name, 'rb')
self.stride = stride

def read(self, len_):
return self.fh.read(self.stride*len_)[::self.stride]

def close(self):
self.fh.close()

class FixedStridedStream(StridedStream):
def __del__(self):
# Prevent IOBase.__del__ frombeing called.
pass

warnings.resetwarnings()
warnings.simplefilter('error')

print('Creating  loosing StridedStream..')
r = StridedStream('/dev/zero')
del r

print('Creating  loosing FixedStridedStream..')
r = FixedStridedStream('/dev/zero')
del r

$ python3 test.py 
Creating  loosing StridedStream..
Creating  loosing FixedStridedStream..
Exception ignored in: _io.FileIO name='/dev/zero' mode='rb'
ResourceWarning: unclosed file _io.BufferedReader name='/dev/zero'

In the first case, the destructor inherited from IOBase actually
prevents the ResourceWarning from being emitted.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subprocess shell=True on Windows doesn't escape ^ character

[Nikolaus Rath]

R. David Murray rdmur...@bitdance.com writes:
 Also notice that using a list with shell=True is using the API
 incorrectly.  It wouldn't even work on Linux, so that torpedoes
 the cross-platform concern already :)

 This kind of confusion is why I opened http://bugs.python.org/issue7839.

Can someone describe an use case where shell=True actually makes sense
at all?

It seems to me that whenever you need a shell, the argument's that you
pass to it will be shell specific. So instead of e.g.

Popen('for i in `seq 42`; do echo $i; done', shell=True)

you almost certainly want to do

Popen(['/bin/sh', 'for i in `seq 42`; do echo $i; done'], shell=False)

because if your shell happens to be tcsh or cmd.exe, things are going to
break.

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why does IOBase.__del__ call .close?

[Nikolaus Rath]

MRAB pyt...@mrabarnett.plus.com writes:
 On 2014-06-11 02:30, Nikolaus Rath wrote:
 Hello,

 I recently noticed (after some rather protacted debugging) that the
 io.IOBase class comes with a destructor that calls self.close():

 [0] nikratio@vostro:~/tmp$ cat test.py
 import io
 class Foo(io.IOBase):
  def close(self):
  print('close called')
 r = Foo()
 del r
 [0] nikratio@vostro:~/tmp$ python3 test.py
 close called

 To me, this came as quite a surprise, and the best documentation of
 this feature seems to be the following note (from the io library
 reference):

 The abstract base classes also provide default implementations of some
   methods in order to help implementation of concrete stream classes. For
   example, BufferedIOBase provides unoptimized implementations of
   readinto() and readline().

 For me, having __del__ call close() does not qualify as a reasonable
 default implementation unless close() is required to be idempotent
 (which one could deduce from the documentation if one tries to, but it's
 far from clear).

 Is this behavior an accident, or was that a deliberate decision?

 To me, it makes sense. You want to make sure that it's closed, releasing
 any resources it might be holding, even if you haven't done so
 explicitly.

I agree with your intentions, but I come to the opposite conclusion:
automatically calling close() in the destructor will hide that there's a
problem in the code. Without that automatic cleanup, there's at least a
good chance that a ResourceWarning will be emitted so the problem gets
noticed. Silently work around bugs in caller's code doesn't seem like
a very useful default to me...


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Why does IOBase.__del__ call .close?

[Nikolaus Rath]

Hello,

I recently noticed (after some rather protacted debugging) that the
io.IOBase class comes with a destructor that calls self.close():

[0] nikratio@vostro:~/tmp$ cat test.py
import io
class Foo(io.IOBase):
def close(self):
print('close called')
r = Foo()
del r
[0] nikratio@vostro:~/tmp$ python3 test.py
close called

To me, this came as quite a surprise, and the best documentation of
this feature seems to be the following note (from the io library
reference):

The abstract base classes also provide default implementations of some
 methods in order to help implementation of concrete stream classes. For
 example, BufferedIOBase provides unoptimized implementations of
 readinto() and readline().

For me, having __del__ call close() does not qualify as a reasonable
default implementation unless close() is required to be idempotent
(which one could deduce from the documentation if one tries to, but it's
far from clear).

Is this behavior an accident, or was that a deliberate decision?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes

[Nikolaus Rath]

Nathaniel Smith n...@pobox.com writes:
 Such optimizations are important enough that numpy operations always
 give the option of explicitly specifying the output array (like
 in-place operators but more general and with clumsier syntax). Here's
 an example small-array benchmark that IIUC uses Jacobi iteration to
 solve Laplace's equation. It's been written in both natural and
 hand-optimized formats (compare num_update to num_inplace):

https://yarikoptic.github.io/numpy-vbench/vb_vb_app.html#laplace-inplace

 num_inplace is totally unreadable, but because we've manually elided
 temporaries, it's 10-15% faster than num_update. 

Does it really have to be that ugly? Shouldn't using

  tmp += u[2:,1:-1]
  tmp *= dy2
  
instead of

  np.add(tmp, u[2:,1:-1], out=tmp)
  np.multiply(tmp, dy2, out=tmp)

give the same performance? (yes, not as nice as what you're proposing,
but I'm still curious).


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [numpy wishlist] Interpreter support for temporary elision in third-party classes

[Nikolaus Rath]

Nathaniel Smith n...@pobox.com writes:
  tmp1 = a + b
  tmp1 += c
  tmp1 /= c
  result = tmp1

 Could this transformation be done in the ast? And would that help?

 I don't think it could be done in the ast because I don't think you can
 work with anonymous temporaries there. But, now that you mention it, it
 could be done on the fly in the implementation of the relevant opcodes.
 I.e., BIN_ADD could do

 if (Py_REFCNT(left) == 1)
 result = PyNumber_InPlaceAdd(left, right);
 else
 result = PyNumber_Add(left, right)

 Upside: all packages automagically benefit!

 Potential downsides to consider:
 - Subtle but real and user-visible change in Python semantics. I'd be a
 little nervous about whether anyone has implemented, say, an iadd with side
 effects such that you can tell whether a copy was made, even if the object
 being copied is immediately destroyed. Maybe this doesn't make sense
 though.

Hmm. I don't think this is as unlikely as it may sound. Consider eg the
h5py module:

with h5py.File('database.h5') as fh:
 result = fh['key'] + np.ones(42)

if this were transformed to

with h5py.File('database.h5') as fh:
tmp = fh['key']
tmp += np.ones(42)
result = tmp

then the database.h5 file would get modified, *and* result would be of
type h5py.Dataset rather than np.array.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Commit-ready patches in need of review

[Nikolaus Rath]

Hello,

While my last appeal resulted in quite some commits (thanks!), I still
have some more commit-ready patches waiting for review.  It'd be great
if some people could find time to take a look:

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)
  
  Reviewed patch, rebased on current hg tip
  
* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

  I only wrote the patch for one file because I'd like to have feedback
  before tackling the second. However, the patches are independent so
  unless there are other problems this is ready for commit.
  
* http://bugs.python.org/issue15955 (gzip, bz2, lzma: add option to
  limit output size)
  
* http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
  missing)

  
Best,
Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Aborting unit tests on first failure

[Nikolaus Rath]

Hello,

I've just run the testsuite of hg tip with

./python -m test -u network,urlfetch -j 8 -G -v

and it finished with

,
| [...]
| test_extract_dir (test.test_zipfile.TestWithDirectory) ... ok
| test_store_dir (test.test_zipfile.TestWithDirectory) ... ok
| test_different_file (test.test_zipfile.TestsWithMultipleOpens) ... ok
| test_interleaved (test.test_zipfile.TestsWithMultipleOpens) ... ok
| test_same_file (test.test_zipfile.TestsWithMultipleOpens) ... ok
| 
| --
| Ran 163 tests in 14.522s
| 
| OK (skipped=25)
| 368 tests OK.
| 2 tests failed:
| test_decimal test_itertools
| 1 test altered the execution environment:
| test___all__
| 17 tests skipped:
| test_bz2 test_curses test_dbm_gnu test_devpoll test_idle
| test_kqueue test_msilib test_ossaudiodev test_readline
| test_startfile test_tcl test_tk test_ttk_guionly test_ttk_textonly
| test_winreg test_winsound test_zipfile64
`

I thought the -G option is would cause the test to stop as soon as an
error occured:

|  -G, --failfastfail as soon as a test fails (only with -v or -W)

But it my case it seems that it actually continued to run all the other
test modules. Did I misunderstand what -G is supposed to do, or is this
a bug in the test runner?

It seems to work fine within a single test module, i.e. if I run

| ./python -m test -u network,urlfetch -j 8 -G -v test_decimal

..then execution stops right after the failed test.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bytes with trailing \0?

[Nikolaus Rath]

Hello,

I was surprised to find the following in bytesobject.c:

,
| [...]
|As always, an extra byte is allocated for a trailing \0 byte (newsize
|does *not* include that), and a trailing \0 byte is stored.
| */
| 
| int
| _PyBytes_Resize(PyObject **pv, Py_ssize_t newsize)
| {
| [...]
| 
`

Does this mean that bytes objects are internally stored with a trailing
\0? Why is that? Isn't that just wasting a byte, because \0 might also
be in the middle of the byte sequence, and the bytes objects stores its
length explicitly anyway?


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Commit-ready patches needing review

[Nikolaus Rath]

Hello,

While my last appeal resulted in quite some commits (thanks!), I still
have some more commit-ready patches waiting for review.  It'd be great
if some people could find time to take a look:

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)

* http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for
  non-blocking socket)

  In this case someone just needs to decide if we want to (a) document
  the current behavior, (b) deprecate the current behavior or (c) change
  the current behavior. I have attached patches for (a) and (b), and if
  (c) is the desired route I'll be happy to create a patch on short
  notice.

* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

* http://bugs.python.org/issue19414 (iter(ordered_dict) yields keys
  not in dict in some circumstances)

* http://bugs.python.org/issue15955 (gzip, bz2, lzma: add option to
  limit output size)

  Nadeem has kindly reviewed the first iteration of this patch, but the
  second iteration has been waiting for attention for quite some time
  now.

* http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
  missing)

* http://bugs.python.org/issue21057 (TextIOWrapper does not support
  reading bytearrays or memoryviews)


Best,
Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Commit-ready patches needing review

[Nikolaus Rath]

Antoine Pitrou solip...@pitrou.net writes:
 On Sun, 27 Apr 2014 12:10:46 -0700
 Nikolaus Rath nikol...@rath.org wrote:
 
 * http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for
   non-blocking socket)
 
   In this case someone just needs to decide if we want to (a) document
   the current behavior, (b) deprecate the current behavior or (c) change
   the current behavior. I have attached patches for (a) and (b), and if
   (c) is the desired route I'll be happy to create a patch on short
   notice.

 In this case I'd be inclined to follow Ben Darnell's advice and change
 the current behaviour (i.e., let the exception bubble up rather than
 catch it). This is what your initial patch does. However, it would need
 a documentation addition to explain the change (and perhaps a test,
 though that doesn't seem terribly necessary here).

Sounds good to me. I just attached an updated patch to the issue.


Thanks for looking at this!


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appeal for reviews

[Nikolaus Rath]

Terry Reedy tjre...@udel.edu writes:
[Quote conveniently rearranged]
 I've accumulated a number of patches in the issue tracker that are
 waiting for someone to review/commit/reject them. I'm eager to make
 corrections as necessary, I just need someone to look the work that I've
 done so far:

 Do you consider any of these 'ready-to-commit'?
 * before-fail, after-pass test?
 * required doc changes?
 * tested patch for all versions that should be patched?
 * any new Misc/ACKS entry needed (for new person other than you)?
 * checked for stray end-of-line whitespace?

Actually, they should all be in this stage (assuming make patchcheck
would complain about eol whitespace). I wouldn't call it ready-to-commit
though, I'm sure a second pair of eyes would find issues that need to be
addressed. But I'm at a point where I don't see any possible issues.

In more detail:

 * http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for
non-blocking socket)

This contains a docpatch for 3.4 that (I believe) should be applied in
any case. In addition, there does not seem to be a consensus whether the
same behavior should be kept for 3.5 (in that case the docpatch should
be applied for 3.5 as well), deprecated in 3.5 (patch available as
well), or changed right away (no patch attached yet, I'm waiting for
consensus). Maybe it would be enough if some more developers could chime
in? I'm not sure how issues like this are typically decided.

 * http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
only)

Contains testcase, docpatch and code-patch reviewed and updated by
me. New feature, so should go only into 3.5 and applies cleanly.

I just updated the patch to include the original author in Misc/ACKS.

 * http://bugs.python.org/issue7776 (http.client.HTTPConnection
tunneling is broken)

Contains testcase, docpatch and code-patch. Applies cleanly to 3.4, and
I just refreshed the patch for 3.5. Not sure if this should be fixed in
2.7?

 * http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
Argument Clinic across 2 files)

No test case and no doc patch necessary (I believe). I have some doubts
about the patch though, see
http://bugs.python.org/issue20177#msg209153. 

 * http://bugs.python.org/issue19414 (iter(ordered_dict) yields keys
not in dict in some circumstances)

There are several different ways to fix this problem. I have created a
patch implementing one of them. It applies cleanly to 3.4 and 3.5 - not
sure why the review link does not show up. I've just attached the same
patch again, maybe that helps.

 * http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
missing)

Contains doc update, test case, and code patch. Applies cleanly to 3.5,
and should not go in older versions (new feature).

Martin v. Löwis has kindly started reviewing this.

 * http://bugs.python.org/issue21057 (TextIOWrapper does not support
reading bytearrays or memoryviews)

Contains test case, and code patch. I don't think the documentation
needs to be updated. Patch applies cleanly to 3.5, and should not go in
older versions (new feature).

Martin v. Löwis has kindly started reviewing this as well :-).


Best,
-Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appeal for reviews

[Nikolaus Rath]

Stephen J. Turnbull step...@xemacs.org writes:
 I apologize for the tone.  I need to go *right* now, and can't fix
 that.  Really, I'm sympathetic and my goal is not just to defend
 python-dev, but to help you get the reviews your work deserves.
 Please read with that in mind.

Will do - but why the rush? Be assured that I would have read the mail
even if it came a day later :-).

 That's an impressive list, but it doesn't tell us what work you've
 done.  There's a checklist in the devguide which should be more
 accurate (sorry about lack of URL), but I imagine it includes the
 following:

  * Has the issue been classified as bug or feature?
  * If bug, is it confirmed?
  * If complex, does it need a PEP?  (probably irrelevant to your patches)
  * Is there agreement about requirements in the issue comments?
  * Is there a patch?
  * Does the design need review?
  * Has relevant documentation been added/updated (including
docstrings, manuals, Whats New at least)?
  * Are there tests that the requirements are satisfied?
  * Are there tests for any regressions that arose in the process of
developing the patch?
  * Have you signed a CA?  (irrelevant to you IIRC)
  * Is the issue status updated to reflect the work done so far?

I've described the status of each bug in more detail in my reply 
Terry, but generally all the issues contain a testcase (i.e., so I
consider them confirmed), do not require a PEP, contain a patch that
needs review and include documentation updates. I have signed the CA,
and I have updated the issue status as much as I can (I don't seem to
have privileges to add/change keywords). 

I have a second list with issues that I worked on that are more nasty,
but I deliberately did not include them in the list.

 It would help in getting reviewer attention to your work if in
 addition to a list of patches you provided an indication of (1) how
 complete the patch is and (2) what review is requested.  That in
 itself is a strong indicator of quality.

All the patches are complete in the sense that I think they can be
committed and would improve Python. Of course, further improvements are
always possible. I'm not sure about (2) -- what different kind of
reviews are there?


 There must be stuff of interest to you happening on twitter, etc.,
 during PyCon.

I'm not quite sure what you're getting at (maybe because I don't use
twitter).

Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Appeal for reviews

[Nikolaus Rath]

Martin v. Löwis mar...@v.loewis.de writes:
 Am 13.04.14 08:36, schrieb Stephen J. Turnbull:

 I admit the tone was biased toward nagging or blaming the victim,
 and again I apologize for causing misunderstanding.  Nikolaus isn't
 wrong for posting here.  My claim is that in current circumstances,
 core-mentorship would be a more *effective* channel because
 
  - core-mentorship is *explicitly* for poking Those Who Can Help
(among other requests for help);

 It would be worth an experiment. I know that I wouldn't have reviewed
 Nikolaus' patches if he had posted to core-mentorship - I'm not a core
 mentor.

This is actually not the first time I sent this (though luckily, the
number of issues did change somewhat since the last attempt). My last
two mails went to core-mentorship, so I figured that maybe this time I
ought to try a different list to get other people's attention as well.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Appeal for reviews

[Nikolaus Rath]

Hello,

I've accumulated a number of patches in the issue tracker that are
waiting for someone to review/commit/reject them. I'm eager to make
corrections as necessary, I just need someone to look the work that I've
done so far:

* http://bugs.python.org/issue20951 (SSLSocket.send() returns 0 for
  non-blocking socket)

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)

* http://bugs.python.org/issue7776 (http.client.HTTPConnection
  tunneling is broken)

* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

* http://bugs.python.org/issue19414 (iter(ordered_dict) yields keys
  not in dict in some circumstances)

* http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
  missing)

* http://bugs.python.org/issue21057 (TextIOWrapper does not support
  reading bytearrays or memoryviews)

  
I realize that core developer time is scarce, so I have started to only
work on patches after I've confirmed that someone is available and
interested to review them. However, it would be great if some people
could find time to look at the issues above as well. Having your
contributions just languish in the bugtracker is really dispiriting... I
*want* to contribute, and I can't believe that Python is the one
open-source project that is not in need of more manpower. But for some
reason it seems really hard to actually find something to do..


Best,
Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Adding a readinto1 method to BufferedReader

[Nikolaus Rath]

Hello,

The BufferedReader (and BufferedRWPair) classes both have a read1()
method in addition to the regular read() method to bypass the internal
buffer. This is quite useful if you need to do some buffered reading
(e.g. to parse a header) followed by a lot of bulk data that you want to
process as it streams in.

However, the readinto() method does not have a corresponding readinto1()
method. I would like to add this method. I have proposed a patch in
http://bugs.python.org/issue20578.

Are there any comments, objections, encouragements...?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSLSocket.send() for non-blocking sockets

[Nikolaus Rath]

Chris Angelico ros...@gmail.com writes:
 On Wed, Mar 26, 2014 at 11:54 AM, Nikolaus Rath nikol...@rath.org wrote:
 2. Change the behavior immediately, potentially breaking some
applications that worked around it, but unbreaking others that relied
on the documented behavior.

 If it's a functionality change that's likely to break code, would it
 be worth changing it only in 3.5, and documenting it as broken in 3.4
 and earlier?

Yes, that's what I meant. I don't think changing it in 3.4 is an option
at all.


Best,
-Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 466 (round 2): Network security enhancements for Python 2.7

[Nikolaus Rath]

Nick Coghlan ncogh...@gmail.com writes:
 Maintainability
 ---

 This policy does NOT represent a commitment by volunteer contributors to
 actually backport network security related changes from the Python 3 series
 to the Python 2 series. Rather, it is intended to send a clear signal to
 potential corporate contributors that the core development team are willing
 to review and merge corporate contributions that put this policy into
 effect.

As I understand, at least for smaller patches it is actually more work
to apply a patch than than to write it. With that in mind, are there
really sufficient volunteer resources available to review and merge
these corporate contributions if they come? The issue tracker certainly
does not lack issues with unreviewed and/or unapplied patches...


Best,
-Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Confirming status of new modules in 3.4

[Nikolaus Rath]

Charles-François Natali cf.nat...@gmail.com writes:
 2014-03-15 21:44 GMT+00:00 Nikolaus Rath nikol...@rath.org:

 Guido van Rossum gu...@python.org writes:
  This downside of using subclassing as an API should be well known by now
  and widely warned against.

 It wasn't known to me until now. Are these downsides described in some
 more detail somewhere?


 The short version is: inheritance breaks encapsulation.

 As a trivial and stupid example, let's say you need a list object which
 counts the number of items inserted/removed (it's completely stupid, but
 that's not the point :-):

 So you might do something like:
[...]

Very illuminating example, thanks a lot!


Best,
-Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Confirming status of new modules in 3.4

[Nikolaus Rath]

Guido van Rossum gu...@python.org writes:
 This downside of using subclassing as an API should be well known by now
 and widely warned against.

It wasn't known to me until now. Are these downsides described in some
more detail somewhere?


So far I have always thought that, as long as I avoid using private
attributes, subclassing was supported like any other use of the API.


Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Requesting pronouncement on PEP 463: Exception-catching expressions

[Nikolaus Rath]

Tres Seaver tsea...@palladion.com writes:
 On 03/12/2014 04:49 PM, Chris Angelico wrote:
 You can use hasattr() in place of AttributeError

 I use:

  getattr(subject, attrname, default)?

 *all the time*.

In my opinion that's almost as ugly, because it still forces you to
specify the attribute name as a string rather than an identifier.


Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2

[Nikolaus Rath]

Ethan Furman et...@stoneleaf.us writes:
 Example::

 b'%4x' % 10
b'   a'

 '%#4x' % 10
' 0xa'

 '%04X' % 10
'000A'

Shouldn't the second two examples also be bytes, ie. b'%#4x' instead of
'%#4x'?


Best,
-Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 463: Exception-catching expressions

[Nikolaus Rath]

Stephen J. Turnbull step...@xemacs.org writes:
 Ethan Furman writes:
   On 02/21/2014 07:46 PM, Chris Angelico wrote:
   
but not this:
   
value = expr except Exception: default except Exception: default
   
   This should be the way it works.  Nothing is gained in readability
   by turning a try with multiple except statements into an
   expression.

 Examples have been given several times.  In general, if 'expr' is a
 function call, it may well have a couple of different ways to fail
 which imply different default values.

 interpolable = func(key) except TypeError: not a string: %s % key \
  except KeyError: no such key: %s % key
 print(Some message that refers to '%s' % interpolable)

 versus

 try:
 interpolable = func(key)
 except TypeError:
 interpolable = not a string: %s % key
 except KeyError:
 interpolable = no such key: %s % key
 print(Some message that refers to '%s' % interpolable)

I think the following suggestion from elsewhere in the thread would look
even better in this case:

 interpolable = func(key) except (TypeError: not a string: %s % key,
  KeyError: no such key: %s % key)
 print(Some message that refers to '%s' % interpolable)

 
It does not require the backslash, it is shorter, and it can still be
chained:

 interpolable = func(key) except (TypeError: not a string: %s % key,
  KeyError: defaults[key]
  except (KeyError: no such key: %s % 
key))
 print(Some message that refers to '%s' % interpolable)

Best,
-Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The docstring hack for signature information has to go

[Nikolaus Rath]

Larry Hastings la...@hastings.org writes:
 In the second attempt, the signature looked like this:

sig=(arguments)\n

[...]
 This all has caused no problems so far.  But my panicky email last
 night was me realizing a problem we may see down the road.  To recap:
 if a programmer writes a module using the binary ABI, in theory they
 can use it with different Python versions without modification.  If
 this programmer added Python 3.4+ compatible signatures, they'd have
 to insert this sig=( line at the top of their docstring.  The
 downside: Python 3.3 doesn't understand that this is a signature and
 would happily display it to the user as part of help().

I think this is not a bug, it's a feature. Since 3.3 users don't have
the special signature parser either, this gives them exactly the
information they need and without any duplication. The only drawback is
in the cosmetic sig= prefix -- but that's the right amount of
non-intrusive, kind nudging to get people to eventually update.

 How bad would it be if we decided to just live with it or if we
 added a new flag bit (only recognized by 3.4) to disambiguate
 corner-cases?

 A new flag might solve the problem cheaply.  Let's call it METH_SIG,
 set in the flags portion of the PyMethodDef.  It would mean This
 docstring contains a computer-readable signature.  One could achieve
 source compatibility with 3.3 easily by adding #ifndef METH_SIG /
 #define METH_SIG 0 / #endif; the next version of 3.3 could add that
 itself.  We could then switch back to the original approach of
 name-of-function(, so the signature would look presentable when
[...]

That much effort to fix a purely cosmetic problem showing up only in
older releases? Note that it's going to be a while until machine
generated signatures have actually trickled down to end-users, so it's
not as if every 3.3 installation would suddenly show different
docstrings for all modules. 

Just my $0.02 of course.

Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Larry Hastings la...@hastings.org writes:
 A comment on your approach so far: I'm very much against giving
 default a default value in the constructor.

You mean in the definition of the custom converter class?

 I realize that hack saves you having to say = NULL in a lot of
 places.  But explicit is better than implicit, and we're going to read
 these signatures a lot more often than we write them, and I want
 Clinic signatures to be easy to read at first glance.
[]
 All is not lost!  What follows is rough pseudo-C code, hopefully you
 can take it from here.

typedef struct {
   int set;
   time_t when;
} clinic_time_t;
#define DEFAULT_CLINIC_TIME_T {0, 0}

[...]

/*[python input]
class clinic_time_t_converter(CConverter):
 type = 'clinic_time_t'
 converter = 'parse_clinic_time_t'
 c_default = 'DEFAULT_CLINIC_TIME_T'
[python start generated code]*/
/*[python end generated code: checksum=...]*/

 Now you can use clinic_time_t.  Parameters declared clinic_time_t can
 be required, or they can be optional; if they're optional give them a
 default value of None.

That doesn't work. If the default value is declared for the function
rather than in the converter definition, it overwrites the C default:

/*[clinic input]
time.gmtime

seconds: clinic_time_t=None
/
*/

gives:

static PyObject *
time_gmtime(PyModuleDef *module, PyObject *args)
{
PyObject *return_value = NULL;
clinic_time_t seconds = Py_None;

if (!PyArg_ParseTuple(args,
|O:gmtime,
parse_clinic_time_t, seconds))
goto exit;
return_value = time_gmtime_impl(module, seconds);

so the default for seconds is now Py_None instead of
DEFAULT_CLINIC_TIME_T'.


Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Larry Hastings la...@hastings.org writes:
 All is not lost!  What follows is rough pseudo-C code, hopefully you
 can take it from here.

typedef struct {
   int set;
   time_t when;
} clinic_time_t;

#define DEFAULT_CLINIC_TIME_T {0, 0}

static int
parse_clinic_time_t_arg(PyObject *arg, clinic_time_t *ct)
{
 if (arg == NULL)
 return 1;
 if (arg == Py_None)
 return 0;
 if (_PyTime_ObjectToTime_t(arg, ct-when) == -1) {
 set = 1;
 return 0;
 }
 return 1;
}

static int post_parse_clinic_time_t(clinic_time_t *ct) {
 if (ct-set)
 return 0;
 ct-when = time(NULL);
 return 0;
}

/*[python input]
class clinic_time_t_converter(CConverter):
 type = 'clinic_time_t'
 converter = 'parse_clinic_time_t'
 c_default = 'DEFAULT_CLINIC_TIME_T'
[python start generated code]*/
/*[python end generated code: checksum=...]*/

 Now you can use clinic_time_t.  Parameters declared clinic_time_t can
 be required, or they can be optional; if they're optional give them a
 default value of None.  You'll have to call post_parse_clinic_time_t()
 by hand in your impl function; I'll see if I can extend Clinic so
 converters can emit code after a successful call to the parse function
 but before the call to the impl.


Okay, I attached a patch along these lines to the bugtracker.


However, I have to say that I lost track of what we're actually gaining
with all this. If all we need is a type that can distinguish between a
valid time_t value and a default value, why don't we simply use
PyObject?

In other words, what's the advantage of all the code above over:

static int
PyObject_to_time_t(PyObject *obj, time_t *when)
{
 if (obj == NULL || obj == Py_None) 
 *when = time(NULL);
 else if (_PyTime_ObjectToTime_t(obj, when) == -1)
 return 0;
 return 1;
}


/*[clinic input]
time.gmtime

seconds: object=NULL
/
[...]

static PyObject *
time_gmtime_impl(PyModuleDef *module, PyObject *seconds)
{
PyObject *return_value = NULL;
time_t when;
if(!PyObj_to_time_t(seconds, when))
   return NULL;

[...]


To me this version looks shorter and clearer. Is there really an
advantage in defining the clinic argument as a custom struct rather than
object?


Best,
-Nikolaus


-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Ryan Smith-Roberts r...@lab.net writes:
 The trick you're missing is that *any time* you have an optional argument
 with a custom converter[1], PyArg_ParseTuple *only* calls the converter
 function in the case that the user *actually supplies some value*. This is
 a basic property of an optional argument. Another property is that the
 c_default is evaluated *every time*, as it is set *before* the call to
 PyArg_ParseTuple. Are these the best ways to do things? Maybe not, but it's
 how they are.

 Please do not use a custom converter for this case. It can't work. 

Well, I thought I was following Larry's recommendation:

 A better choice would be to write a converter function in C, then use
 a custom converter that called it.  Nikolaus: Is that something you're
 comfortable doing?

..and I assumed he'd know best. Did I misunderstand that quote?

 Please do what I outlined earlier (untested, somewhat verbose code
 follows):

 static int
 parse_time_t_arg(PyObject *arg, time_t *when)
 {
 if (arg == NULL || arg == Py_None) {
 *when = time(NULL);
 return 1;
 }
 if (_PyTime_ObjectToTime_t(arg, when) == -1)
 return 0;
 return 1;
 }

Ahm, this is exactly the code that I wrote (and you even quoted it
below), only with the identifiers renamed.

 /*[clinic input]
 time.gmtime
 seconds: object = None
 [clinic start generated code]*/
 {
 time_t when;

 if (0 == parse_time_t_arg(seconds, when))
 return NULL;


That's fine with me too. I'd just like Larry to sign off on it, because
as far as I know, he'll be the one to review my patch.


Best,
-Nikolaus

 [1] If you set a default value, or put it in brackets as Serhiy later
 recommends, it works the same.


 On Sun, Jan 19, 2014 at 8:19 PM, Nikolaus Rath nikol...@rath.org wrote:

 Larry Hastings la...@hastings.org writes:
  On 01/18/2014 09:52 PM, Ryan Smith-Roberts wrote:
 
  I still advise you not to use this solution. time() is a system call
  on many operating systems, and so it can be a heavier operation than
  you'd think. Best to avoid it unless it's needed (on FreeBSD it
  seems to add about 15% overhead to localtime(), for instance).
 
 
  I agree.  Converting to Argument Clinic should not cause a performance
  regression.  Please don't add new calls to time() for the sake of
  making code more generic.
 
  A better choice would be to write a converter function in C, then use
  a custom converter that called it.  Nikolaus: Is that something you're
  comfortable doing?

 I think I'll need some help. I don't know how to handle the case where
 the user is not passing anything.

 Here's my attempt:

 ,
 | /* C Converter for argument clinic
 |If obj is NULL or Py_None, return current time. Otherwise,
 |convert Python object to time_t.
 | */
 | static int
 | PyObject_to_time_t(PyObject *obj, time_t *stamp)
 | {
 | if (obj == NULL || obj == Py_None) {
 | *stamp = time(NULL);
 | }
 | else {
 | if (_PyTime_ObjectToTime_t(obj, stamp) == -1)
 | return 0;
 | }
 | return 1;
 | }
 |
 | /*[python input]
 | class time_t_converter(CConverter):
 | type = 'time_t'
 | converter = 'PyObject_to_time_t'
 | default = None
 | [python start generated code]*/
 | /*[python end generated code:
 checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
 |
 |
 | /*[clinic input]
 | time.gmtime
 |
 | seconds: time_t
 | /
 |
 | [clinic start generated code]*/
 `

 but this results in the following code:

 ,
 | static PyObject *
 | time_gmtime(PyModuleDef *module, PyObject *args)
 | {
 | PyObject *return_value = NULL;
 | time_t seconds;
 |
 | if (!PyArg_ParseTuple(args,
 | |O:gmtime,
 | PyObject_to_time_t, seconds))
 | goto exit;
 | return_value = time_gmtime_impl(module, seconds);
 |
 | exit:
 | return return_value;
 | }
 `

 This works if the user calls time.gmtime(None), but it fails for
 time.gmtime(). It seems that in that case my C converter function is
 never called.

 What's the trick that I'm missing?


 Thanks!
 -Nikolaus

 --
 Encrypted emails preferred.  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6
 02CF A9AD B7F8 AE4E 425C

  »Time flies like an arrow, fruit flies like a Banana.«
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/rmsr%40lab.net



-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Serhiy Storchaka storch...@gmail.com writes:
 20.01.14 06:19, Nikolaus Rath написав(ла):
 This works if the user calls time.gmtime(None), but it fails for
 time.gmtime(). It seems that in that case my C converter function is
 never called.

 What's the trick that I'm missing?

 /*[clinic input]
 time.gmtime

 [
 seconds: time_t
 ]
 /


Ahh, interesting. So this works, but now the C default is evaluated even
if the user passed an argument (so that answers my question about
decreased performance in the other subthread). The generated code is:

time_gmtime(PyModuleDef *module, PyObject *args)
{
PyObject *return_value = NULL;
int group_right_1 = 0;
time_t seconds = time(NULL);

switch (PyTuple_GET_SIZE(args)) {
case 0:
break;
case 1:
if (!PyArg_ParseTuple(args, O:gmtime, PyObject_to_time_t, 
seconds))
return NULL;
group_right_1 = 1;
break;
default:
PyErr_SetString(PyExc_TypeError, time.gmtime requires 0 to 1 
arguments);
return NULL;
}
return_value = time_gmtime_impl(module, group_right_1, seconds);


All in all, I'm still not sure how I'm supposed to proceed. I see the
following options (and I'm fine with all of them):

1. Use the option group with a custom converter. This means a time(NULL)
   call even if the caller passed a parameter.

2. Declare the _impl parameter as PyObject* instead of time_t, and
   explicitly call a C conversion function.
   
3. Patch clinic.py to only evaluate the C default if the caller does not
   pass a parameter. This seemest cleanest, but I don't know if the
   design of clinic.py actually allows that.


   
Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Larry Hastings la...@hastings.org writes:
 On 01/18/2014 09:52 PM, Ryan Smith-Roberts wrote:

 I still advise you not to use this solution. time() is a system call
 on many operating systems, and so it can be a heavier operation than
 you'd think. Best to avoid it unless it's needed (on FreeBSD it
 seems to add about 15% overhead to localtime(), for instance).


 I agree.  Converting to Argument Clinic should not cause a performance
 regression.  Please don't add new calls to time() for the sake of
 making code more generic.

I don't see how the conversion would result in more calls to time() than
we have now. It seems to me that the expression for the C default should
be only evaluated if the caller did not specify a value. Is that not how
ac works?

 A better choice would be to write a converter function in C, then use
 a custom converter that called it.  Nikolaus: Is that something you're
 comfortable doing?

As long as you're comfortable looking over the (probably buggy) patch,
yes, I'm happy to give it a shot.

Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Larry Hastings la...@hastings.org writes:
 On 01/18/2014 09:52 PM, Ryan Smith-Roberts wrote:

 I still advise you not to use this solution. time() is a system call
 on many operating systems, and so it can be a heavier operation than
 you'd think. Best to avoid it unless it's needed (on FreeBSD it
 seems to add about 15% overhead to localtime(), for instance).


 I agree.  Converting to Argument Clinic should not cause a performance
 regression.  Please don't add new calls to time() for the sake of
 making code more generic.

 A better choice would be to write a converter function in C, then use
 a custom converter that called it.  Nikolaus: Is that something you're
 comfortable doing?

I think I'll need some help. I don't know how to handle the case where
the user is not passing anything.

Here's my attempt:

,
| /* C Converter for argument clinic
|If obj is NULL or Py_None, return current time. Otherwise,
|convert Python object to time_t.
| */
| static int
| PyObject_to_time_t(PyObject *obj, time_t *stamp)
| {
| if (obj == NULL || obj == Py_None) {
| *stamp = time(NULL);
| }
| else {
| if (_PyTime_ObjectToTime_t(obj, stamp) == -1)
| return 0;
| }
| return 1;
| } 
| 
| /*[python input]
| class time_t_converter(CConverter):
| type = 'time_t'
| converter = 'PyObject_to_time_t'
| default = None
| [python start generated code]*/
| /*[python end generated code: 
checksum=da39a3ee5e6b4b0d3255bfef95601890afd80709]*/
| 
| 
| /*[clinic input]
| time.gmtime
| 
| seconds: time_t
| /
| 
| [clinic start generated code]*/
`

but this results in the following code:

,
| static PyObject *
| time_gmtime(PyModuleDef *module, PyObject *args)
| {
| PyObject *return_value = NULL;
| time_t seconds;
| 
| if (!PyArg_ParseTuple(args,
| |O:gmtime,
| PyObject_to_time_t, seconds))
| goto exit;
| return_value = time_gmtime_impl(module, seconds);
| 
| exit:
| return return_value;
| }
`

This works if the user calls time.gmtime(None), but it fails for
time.gmtime(). It seems that in that case my C converter function is
never called.

What's the trick that I'm missing?


Thanks!
-Nikolaus

-- 
Encrypted emails preferred.  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6
02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Hello,

I'm trying to convert functions using parse_time_t_args() (from
timemodule.c) for argument parsing to argument clinic.

The function is defined as:

,
| static int
| parse_time_t_args(PyObject *args, char *format, time_t *pwhen)
| {
| PyObject *ot = NULL;
| time_t whent;
| 
| if (!PyArg_ParseTuple(args, format, ot))
| return 0;
| if (ot == NULL || ot == Py_None) {
| whent = time(NULL);
| }
| else {
| if (_PyTime_ObjectToTime_t(ot, whent) == -1)
| return 0;
| }
| *pwhen = whent;
| return 1;
| }
`

and used like this:

,
| static PyObject *
| time_localtime(PyObject *self, PyObject *args)
| {
| time_t when;
| struct tm buf;
| 
| if (!parse_time_t_args(args, |O:localtime, when))
| return NULL;
| if (pylocaltime(when, buf) == -1)
| return NULL;
| return tmtotuple(buf);
| }
`

In other words, if any Python object is passed to it, it calls
_PyTime_ObjectToTime_t on it to convert it to time_t, and otherwise uses
time(NULL) as the default value.

May first attempt to implement something similar in argument clinic was:

,
| /*[python input]
| class time_t_converter(CConverter):
| type = 'time_t'
| converter = 'time_t_converter'
| default = None
| py_default = 'None'
| c_default = 'time(NULL)'
| converter = '_PyTime_ObjectToTime_t'
| [python start generated code]*/
| 
| /*[clinic input]
| time.localtime
| 
| seconds: time_t
| /
| 
| bla.
| [clinic start generated code]*/
`

However, running clinic.py on this file gives:

,
| $ Tools/clinic/clinic.py Modules/timemodule.c 
| Error in file Modules/timemodule.c on line 529:
| Exception raised during parsing:
| Traceback (most recent call last):
|   File Tools/clinic/clinic.py, line 1445, in parse
| parser.parse(block)
|   File Tools/clinic/clinic.py, line 2738, in parse
| self.state(None)
|   File Tools/clinic/clinic.py, line 3468, in state_terminal
| self.function.docstring = self.format_docstring()
|   File Tools/clinic/clinic.py, line 3344, in format_docstring
| s += .join(a)
| TypeError: sequence item 2: expected str instance, NoneType found
`

What am I doing wrong?


Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Using argument clinic to replace timemodule.c:parse_time_t_args()

[Nikolaus Rath]

Hi Ryan,


Ryan Smith-Roberts r...@lab.net writes:
 Hi Nikolaus. I also started a conversion of timemodule, but dropped it when
 I saw in the issue that you had taken over that conversion. I also tried to
 turn parse_time_t_args into a converter. However, it won't work. The
 problem is that parse_time_t_args must be called whether or not the user
 supplies an argument to the function, but an Argument Clinic converter only
 gets called if the user actually supplies something, and not on the default
 value.

I don't quite follow. My approach was to drop parse_time_t_args()
completely and use _PyTime_ObjectToTime_t() as the conversion function
(which only needs to be called if the user supplied something).

In other words, I would have expected

 ,
 | /*[python input]
 | class time_t_converter(CConverter):
 | type = 'time_t'
 | converter = 'time_t_converter'
 | default = None
 | py_default = 'None'
 | c_default = 'time(NULL)'
 | converter = '_PyTime_ObjectToTime_t'
 | [python start generated code]*/
 |
 | /*[clinic input]
 | time.localtime
 |
 | seconds: time_t
 | /
 |
 | bla.
 | [clinic start generated code]*/
 `

to produce something like this:

static PyObject *
time_localtime(PyObject *self, PyObject *args)
{
 PyObject *obj = NULL;
 time_t seconds;
 struct tm buf;

 if (!PyArg_ParseTuple(args, |O:localtime, obj))
 return NULL;
 if (obj == NULL || obj == Py_None)
 seconds = time(NULL);
 else {
 if (_PyTime_ObjectToTime_t(obj, seconds) == -1)
 return NULL;
 }
 return time_localtime_impl(self, seconds);
}


Apart from getting an error from clinic.py, it seems to me that this
should in principle be possible.

Best,
Nikolaus



 So, the best idea is to

 * Remove the PyArgs_ParseTuple code from parse_time_t_args
 * Declare seconds as a plain object in Argument Clinic
 * Call the modified parse_time_t_args on seconds first thing in the _impl
 functions


 On Sat, Jan 18, 2014 at 4:56 PM, Nikolaus Rath nikol...@rath.org wrote:

 Hello,

 I'm trying to convert functions using parse_time_t_args() (from
 timemodule.c) for argument parsing to argument clinic.

 The function is defined as:

 ,
 | static int
 | parse_time_t_args(PyObject *args, char *format, time_t *pwhen)
 | {
 | PyObject *ot = NULL;
 | time_t whent;
 |
 | if (!PyArg_ParseTuple(args, format, ot))
 | return 0;
 | if (ot == NULL || ot == Py_None) {
 | whent = time(NULL);
 | }
 | else {
 | if (_PyTime_ObjectToTime_t(ot, whent) == -1)
 | return 0;
 | }
 | *pwhen = whent;
 | return 1;
 | }
 `

 and used like this:

 ,
 | static PyObject *
 | time_localtime(PyObject *self, PyObject *args)
 | {
 | time_t when;
 | struct tm buf;
 |
 | if (!parse_time_t_args(args, |O:localtime, when))
 | return NULL;
 | if (pylocaltime(when, buf) == -1)
 | return NULL;
 | return tmtotuple(buf);
 | }
 `

 In other words, if any Python object is passed to it, it calls
 _PyTime_ObjectToTime_t on it to convert it to time_t, and otherwise uses
 time(NULL) as the default value.

 May first attempt to implement something similar in argument clinic was:

 ,
 | /*[python input]
 | class time_t_converter(CConverter):
 | type = 'time_t'
 | converter = 'time_t_converter'
 | default = None
 | py_default = 'None'
 | c_default = 'time(NULL)'
 | converter = '_PyTime_ObjectToTime_t'
 | [python start generated code]*/
 |
 | /*[clinic input]
 | time.localtime
 |
 | seconds: time_t
 | /
 |
 | bla.
 | [clinic start generated code]*/
 `

 However, running clinic.py on this file gives:

 ,
 | $ Tools/clinic/clinic.py Modules/timemodule.c
 | Error in file Modules/timemodule.c on line 529:
 | Exception raised during parsing:
 | Traceback (most recent call last):
 |   File Tools/clinic/clinic.py, line 1445, in parse
 | parser.parse(block)
 |   File Tools/clinic/clinic.py, line 2738, in parse
 | self.state(None)
 |   File Tools/clinic/clinic.py, line 3468, in state_terminal
 | self.function.docstring = self.format_docstring()
 |   File Tools/clinic/clinic.py, line 3344, in format_docstring
 | s += .join(a)
 | TypeError: sequence item 2: expected str instance, NoneType found
 `

 What am I doing wrong?


 Best,
 Nikolaus

 --
 Encrypted emails preferred.
 PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

  »Time flies like an arrow, fruit flies like a Banana.«
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/rmsr%40lab.net



-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana

Re: [Python-Dev] OrderedDict.values() behavior for modified instance

[Nikolaus Rath]

Ethan Furman et...@stoneleaf.us writes:
 Nikolaus,

 Good write-up.  Please submit it to the bug tracker:
 http://bugs.python.org

Submitted as http://bugs.python.org/issue19414.

If someone gives me the go-ahead for one of the proposed solutions, I'd
be happy to create a full patch.

Best,
Nikolaus

-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] OrderedDict.values() behavior for modified instance

[Nikolaus Rath]

Hello,

The documentation says the following about modifying a dict while
iterating through its view:

| Iterating views while adding or deleting entries in the dictionary may
| raise a RuntimeError or fail to iterate over all entries.
(http://docs.python.org/3/library/stdtypes.html#dict-views)

The OrderedDict documentation doesn't have anything to say on the
subject. In practice, its implementation actually mostly correponds to
naive expectations:

 from collections import OrderedDict
 d = OrderedDict()
 for i in range(5):
...d[i] = i
... 
 i = iter(d.values())
 next(i)
0
 del d[0]
 next(i)
1
 del d[2]
 next(i)
3
 d.move_to_end(1)
 next(i)
4
 next(i)
1
 

etc.

However, there is one case that results in a rather confusing error:

 a = OrderedDict()
 a[0] = 0
 a[1] = 1
 a[2] = 2
 i = iter(a.values())
 next(i)
0
 del a[0]
 del a[1]
 next(i)
Traceback (most recent call last):
  File stdin, line 1, in module
  File /usr/lib/python3.3/collections/abc.py, line 495, in __iter__
yield self._mapping[key]
KeyError: 1


What happens here is that a[0] is returned from the linked list, but it
still contains links to a[1]. When a[1] is deleted, the links of its
predecessor and successor are updated, but a[0] still points to
a[1]. The OrderedList then hands a non-existing key to the values()
implementation in collections.abc.

The result is not only mightily confusing, but it is also not covered by
the documentation (KeyError isn't a RuntimeError).


I would very much like to see this fixed, but I'm not sure if there's a
good way to do that.

I see the following options:

(a) When deleting an element from an OrderedList, update the pointers in
the corresponding linked list element to the end of the list. Then
removing the currently active element during the iteration would
automatically end the iteration.

For that, we'd just have to add two lines to __delitem__:

def __delitem__(self, key, dict_delitem=dict.__delitem__):
dict_delitem(self, key)
link = self.__map.pop(key)
link_prev = link.prev
link_next = link.next
link_prev.next = link_next
link_next.prev = link_prev

link.prev = root # new 
link.next = root # new


The new behavior is explicitly covered by the documentation
(changing the dict may result in incomplete iteration), but it's a
change to what happened before.

(b) When iterating through an OrderedDict, check that the next element
is actually still in the hash, i.e. change

def __iter__(self):
root = self.__root
curr = root.next
while curr is not root:
yield curr.key
curr = curr.next
to

def __iter__(self):
root = self.__root
curr = root.next
while curr is not root and curr.key in self:
yield curr.key
curr = curr.next

that would terminate the iteration only in the special case, but
incur an extra dict lookup at every iteration.

Alternatively, one could try very hard to not stop the iteration:

while curr is not root:
yield curr.key
while curr is not root:
curr = curr.next
if curr.key in self:
break

(c) Catch the KeyError in values(), and re-raise the proper exception in
class ValuesView:

def __iter__(self):
for key in self._mapping:
try:
yield self._mapping[key]
except KeyError:
raise RuntimeError(underlying Mapping instance modified during 
iteration)


I consider this a bit ugly, because it may mask other problems in a
custom Mapping implementation (that may raise a KeyError because of
a bug in the Mapping implementation itself).

(d) Extend the OrderedDict documentation to explicity document this
case.

This has the drawback that it would probably be nicer to just have
the behavior be consistent and correspond to intuitive expectations.


Would any of those fixes be acceptable? Or is there an even better option?


Best,
Nikolaus


-- 
Encrypted emails preferred.
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

 »Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 435 - requesting pronouncement

[Nikolaus Rath]

Guido van Rossum gu...@python.org writes:
 1. Having to enter the values is annoying. Sorry, I read the rationale and
 all that, and I *still* want to write a C-Like enum { A, B, C }. I fully
 expect to edit and reorder enums (if I ever use them) and get irritated with
 having to update the value assignments.

 I guess there are cultural differences around this. Anyway, you can
 use the functional/convenience API for this purpose.

Would it be wise to forbid ... as an enum value to preserve the option
to use it for automatic value assignment in some indefinite future?


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] enum instances

[Nikolaus Rath]

On 04/30/2013 07:05 PM, Nikolaus Rath wrote:
 Larry Hastings la...@hastings.org writes:
 On 04/29/2013 07:42 PM, Nikolaus Rath wrote:
 State is a class, it just inherits from enum. Thus:

 type(State) == type(enum) == type(EnumMetaclass)
 issubclass(State, enum) == True


 If you'd tried it, you'd have found that that isn't true.  enum has a
 metaclass, EnumMetaclass.  Thus type(enum) == EnumMetaClass.
 
 That is exactly what I wrote above.

Sorry, I must have read what I thought rather than what I wrote. You're
right, what I wrote was wrong.


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Nikolaus Rath]

Armin Rigo ar...@tunes.org writes:
 Hi Jeff,

 On Mon, Apr 29, 2013 at 11:58 PM, Jeff Allen ja...py@farowl.co.uk wrote:
 In Jython, (...)

 Thanks Jeff for pointing this out.  Jython thus uses a custom
 mechanism similar to PyPy's, which is also similar to atexit's.  It
 should not be too hard to implement it in CPython 3 as well, if this
 ends up classified as a bug.  This is what my bug report was about
 (sorry if I failed to be clear enough about it).

Personally, I think it should just be mentioned in the documentation for
the buffered writers. Otherwise it's hard to justify what deserves such
a special mechanism and what doesn't (what about e.g.
tempfile.NamedTemporaryFile).


 Nikolaus: the bug report contains a failing test, is that what you're
 looking for?

That's what I was trying to write as well, yes. Now I know how to do it
:-)


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Nikolaus Rath]

Armin Rigo ar...@tunes.org writes:
 Hi Nikolaus,

 On Sat, Apr 27, 2013 at 4:39 AM, Nikolaus Rath nikol...@rath.org wrote:
 It's indeed very informative, but it doesn't fully address the question
 because of the _pyio module which certainly can't use any custom C code.
 Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose
 data in the write buffer when the interpreter exits without me calling
 x.close(), but when using x = io.BufferedWriter(), the buffer is
 guaranteed to get flushed?

 I actually described the behavior of CPython 2 while not realizing
 that CPython 3 silently dropped this guarantee.  (I also never
 realized that Jython/IronPython don't have the same guarantee; they
 could, if they implement 'atexit', like we did in PyPy.  That's
 however more acceptable if the platform itself doesn't offer the
 guarantee.)

 Anyway, it's a guarantee that the C offers, so personally I find it
 reasonable to expect CPython files to offer it too; not offering it is
 kind of saying that there is a feature of C that is actually present
 at a higher level than the exact same feature in Python, which looks
 backward to me.

 Additionally, this might be introducing subtle bugs in programs when
 porting them to Python 3.

 However I realize that the two arguments presented above might not be
 accepted as relevant.  (http://bugs.python.org/issue17852)

Thanks for the research! I tried writing a test for this myself, but
failed to get the cyclic reference in place properly for it to happen.

Seems I'm not the only one who was unaware and/or surprised by he change
in behavior.


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] enum instances

[Nikolaus Rath]

Marco Hemmelrath marco.hemmelr...@googlemail.com writes:
 class State(enum):
 idle = 0
 busy = 1
 idling = idle
 ideling = 0

 together with the premises:

 1. type(State.busy) == State
 2. type(State) == enum

State is a class, it just inherits from enum. Thus:

type(State) == type(enum) == type(EnumMetaclass)
issubclass(State, enum) == True


HTH,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library

[Nikolaus Rath]

Steven D'Aprano st...@pearwood.info writes:
 I'm sorry, but all these suggestions are getting the API completely
 backwards by making the common case harder than the rare case.

 We're creating an Enum, right? So the *common case* is to populate it
 with enum values. 99% of the time, enumerated values will be all that
 we want from an enum. So that's the case that needs to be simple, not
 the rare case where you have a non enum value in an enum class.

 The common case (enum values in an Enum class) should be easy, and the
 rare cases (ordinary class-like attributes) possible.

 Explicit is better than implicit: if you want something to *not* be
 processed by the Enum metaclass, you have to explicitly mark it as
 special. Dunders excepted, because they *are* special enough to break
 the rules. Since dunders are reserved for Python, I'm happy with a
 rule that says that dunders cannot be set as enum values (at least not
 via the metaclass). Otherwise, everything inside an Enum class is
 treated as an enum value unless explicitly flagged as not.

 Here's a dirty hack that demonstrates what I'm talking about.
[...]
 class Example(metaclass=MetaEnum):
 red = 1
 blue = 2
 green = lambda: 'good lord, even functions can be enums!'
 def __init__(self, count=3):
 self.count = count
 food = skip('spam')
 @skip
 def spam(self):
 return self.count * self.food


However, without knowing that the MetaEnum metaclass will do some magic
here, there's no way to know that there's anything special about red,
blue and green. So I think there's actually a lot of implicit stuff
happening here.

In contrast,

class Example(metaclass=MetaEnum):
 red = EnumValue(1)
 blue = EnumValue(2)
 green = EnumValue(lambda: 'good lord, even functions can be
 enums!')
 
 def __init__(self, count=3):
 self.count = count

 def spam(self):
 return self.count * self.food

Makes it very clear that red, blue will not be attributes of type int,
even if one has never heard of Enums or metaclasses before.

I don't think this syntax is making the common case hard. By the same
logic, you'd need to introduce C-style i++ postincrement because having
just i += x makes the common case with x=1 hard as well.


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library

[Nikolaus Rath]

Guido van Rossum gu...@python.org writes:
 On Sat, Apr 27, 2013 at 10:04 AM, Ethan Furman et...@stoneleaf.us wrote:
 While this will certainly work, it means you can't have class variables that
 happen to be the same type as the enum -- so no int in an IntEnum, for
 example.

 The solution I like best is the helper class (called, originally enough,
 enum), and only those items get transformed:

 class Planet(IntEnum):
 MERCURY = enum(1)
 VENUS = enum(2)
 EARTH = enum(3)
 rough_pi = 3 # not transformed

 If this means that the most plain vanilla enum definition still has to
 use the enum(i) notation, I'm against it.

I think this is actually a big advantage. It makes it obvious that
something special is going on without having to know that IntEnum uses a
special metaclass.


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Nikolaus Rath]

Guido van Rossum gu...@python.org writes:
 On Monday, April 15, 2013, Nikolaus Rath wrote:
 Brian Curtin br...@python.org javascript:; writes:
  On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath 
  nikol...@rath.orgjavascript:;
 wrote:
  [ Note: I already asked this on
  http://stackoverflow.com/questions/15917502 but didn't get any
  satisfactory answers]
 
  Sorry, but that's not a reason to repost your question to this list.
  If you have to ask somewhere else, it would be python-list, aka,
  comp.lang.python.

 I figured it belonged here because the question is really about the
 internal implementation of file objects, which to me didn't seem like a
 question about using Python. But I'll give it a few days and send
 another mail there if still haven't found the answer by then.

 You got your answer 16 hours ago on S.O.

I guess you are referring to http://stackoverflow.com/a/15968516/293003
from Armin Ringo?

,
| On Windows, NamedTemporaryFile uses a Windows-specific extension
| (os.O_TEMPORARY) to ensure that the file is deleted when it is closed.
| This probably also works if the process is killed in any way. However
| there is no obvious equivalent on POSIX, most likely because on POSIX
| you can simply delete files that are still in use; it only deletes the
| name, and the file's content is only removed after it is closed (in any
| way). But indeed assuming that we want the file name to persist until
| the file is closed, like with NamedTemporaryFile, then we need magic.
| 
| We cannot use the same magic as for flushing buffered files. What occurs
| there is that the C library handles it (in Python 2): the files are FILE
| objects in C, and the C guarantees that they are flushed on normal
| program exit (but not if the process is killed). In the case of Python
| 3, there is custom C code to achieve the same effect. But it's specific
| to this use case, not anything directly reusable.
[...]
`

It's indeed very informative, but it doesn't fully address the question
because of the _pyio module which certainly can't use any custom C code.
Does that mean that when I'm using x = _pyio.BufferedWriter(), I could loose
data in the write buffer when the interpreter exits without me calling
x.close(), but when using x = io.BufferedWriter(), the buffer is
guaranteed to get flushed?

(Note: this isn't a complaint, just curiosity about the Python
internals).


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library

[Nikolaus Rath]

Steven D'Aprano st...@pearwood.info writes:
 On 26/04/13 13:22, Greg wrote:
 On 26/04/2013 3:12 p.m., Glenn Linderman wrote:
 On 4/25/2013 7:49 PM, Nick Coghlan wrote:

 You couldn't create an enum of callables, but that would be a
 seriously weird thing to do anyway

 But aren't all classes callable?

 An enum of classes would be seriously weird as well, I think.


 I don't think iscallable will work, since that descriptors like
 staticmethod and classmethod aren't callable. Nor are properties.


 I think a solution may be an explicit decorator that tells the
 metaclass not to skip the object into an enum value:


 class Insect(enum.Enum):
 ant = 1
 bee = 2

 @enum.skip
 @classmethod
 def spam(cls, args):
 pass


In this case, wouldn't it be nicer to decorate those attributes that
are meant to be enum values? I think having to use the class keyword to
define something that really doesn't behave like an ordinary class is
pretty confusing, and the following syntax would be a lot easier to
understand at first sight:

class Insect(enum.Enum):
ant = enum.EnumValue(1)
bee = enum.EnumValue(2)

@classmethod
def spam(cls, args):
 pass

def ham(self, args):
 pass


Obviously, EnumValue() would automatically assign a suitable number.


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Destructors and Closing of File Objects

[Nikolaus Rath]

Brian Curtin br...@python.org writes:
 On Fri, Apr 12, 2013 at 12:04 AM, Nikolaus Rath nikol...@rath.org wrote:
 [ Note: I already asked this on
 http://stackoverflow.com/questions/15917502 but didn't get any
 satisfactory answers]

 Sorry, but that's not a reason to repost your question to this list.
 If you have to ask somewhere else, it would be python-list, aka,
 comp.lang.python.

I figured it belonged here because the question is really about the
internal implementation of file objects, which to me didn't seem like a
question about using Python. But I'll give it a few days and send
another mail there if still haven't found the answer by then.

Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Destructors and Closing of File Objects

[Nikolaus Rath]

[ Note: I already asked this on
http://stackoverflow.com/questions/15917502 but didn't get any
satisfactory answers]

Hello,

The description of tempfile.NamedTemporaryFile() says:

,
|  If delete is true (the default), the file is deleted as soon as it is
|  closed.
`

In some circumstances, this means that the file is not deleted after the
program ends. For example, when running the following test under
py.test, the temporary file remains:

,
| from __future__ import division, print_function, absolute_import
| import tempfile
| import unittest2 as unittest
| class cache_tests(unittest.TestCase):
| def setUp(self):
| self.dbfile = tempfile.NamedTemporaryFile()
| def test_get(self):
| self.assertEqual('foo', 'foo')
`

In some way this makes sense, because this program never explicitly
closes the file object. The only other way for the object to get closed
would presumably be in the __del__ destructor, but here the language
references states that It is not guaranteed that __del__() methods are
called for objects that still exist when the interpreter exits. So
everything is consistent with the documentation so far.

However, I'm confused about the implications of this. If it is not
guaranteed that file objects are closed on interpreter exit, can it
possibly happen that some data that was successfully written to a
(buffered) file object is lost even though the program exits gracefully,
because it was still in the file objects buffer and the file object
never got closed?

Somehow that seems very unlikely and un-pythonic to me, and the open()
documentation doesn't contain any such warnings either. So I
(tentatively) conclude that file objects are, after all, guaranteed to
be closed.

But how does this magic happen, and why can't NamedTemporaryFile() use
the same magic to ensure that the file is deleted? 


Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com