Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Tim Delaney]

On 9 September 2016 at 15:34, Benjamin Peterson  wrote:

> On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote:
> > On 9 September 2016 at 07:45, Chris Angelico  wrote:
> >
> > > On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner <
> victor.stin...@gmail.com>
> > > wrote:
> > > > A nice "side effect" of compact dict is that the dictionary now
> > > > preserves the insertion order. It means that keyword arguments can
> now
> > > > be iterated by their creation order:
> > > >
> > >
> > > This is pretty sweet! Of course, there are going to be 1172 complaints
> > > from people who's doctests have been broken, same as when hash
> > > randomization came in, but personally, I don't care. Thank you for
> > > landing this!
> > >
> >
> > Are sets also ordered by default now? None of the PEPs appear to mention
> > it.
>
> No.
>

That's an unfortunate inconsistency - I can imagine a lot of people making
the assumption that if dict is ordered (esp. if documented as such) then
sets would be as well. Might need a big red warning in the docs that it's
not the case.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Tim Delaney]

On 9 September 2016 at 07:45, Chris Angelico  wrote:

> On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner 
> wrote:
> > A nice "side effect" of compact dict is that the dictionary now
> > preserves the insertion order. It means that keyword arguments can now
> > be iterated by their creation order:
> >
>
> This is pretty sweet! Of course, there are going to be 1172 complaints
> from people who's doctests have been broken, same as when hash
> randomization came in, but personally, I don't care. Thank you for
> landing this!
>

Are sets also ordered by default now? None of the PEPs appear to mention it.

Tim Delaney
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Benjamin Peterson]

On Thu, Sep 8, 2016, at 22:33, Tim Delaney wrote:
> On 9 September 2016 at 07:45, Chris Angelico  wrote:
> 
> > On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner 
> > wrote:
> > > A nice "side effect" of compact dict is that the dictionary now
> > > preserves the insertion order. It means that keyword arguments can now
> > > be iterated by their creation order:
> > >
> >
> > This is pretty sweet! Of course, there are going to be 1172 complaints
> > from people who's doctests have been broken, same as when hash
> > randomization came in, but personally, I don't care. Thank you for
> > landing this!
> >
> 
> Are sets also ordered by default now? None of the PEPs appear to mention
> it.

No.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Review request: issue 27350, compact ordered dict

[INADA Naoki]

Thank you for all core devs!

I'll polish the implementation until 3.6b2.

On Fri, Sep 9, 2016 at 6:42 AM, Guido van Rossum  wrote:
> It's in! Congrats, and thanks for your great work! See longer post by Victor.
>
> On Sun, Aug 28, 2016 at 12:16 AM, INADA Naoki  wrote:
>> On Sun, Aug 28, 2016 at 2:05 PM, Guido van Rossum  wrote:
>>> Hopefully some core dev(s) can work on this during the core sprint, which is
>>> from Sept 5-9.
>>>
>>
>> OK. While I'm in Japan (UTC+9) and cannot join the sprint, I'll be
>> active as possible
>> while the sprint.
>>
>> Thank you!
>>
>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>
>> --
>> INADA Naoki  
>
>
>
> --
> --Guido van Rossum (python.org/~guido)



-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] IMPORTANT: 3.6.0b1 and feature code cutoff 2016-09-12 12:00 UTC

[Ned Deily]

Happy end of summer (northern hemisphere) or winter (southern)!

Along with the changing of the seasons, the time has come to finish feature 
development for Python 3.6.  As previously announced, this coming Monday marks 
the end of the alpha phase of the release cycle and the beginning of the beta 
phase.  Up through the alpha phase, there has been unrestricted feature 
development phase; that ends as of beta 1.  All feature code for 3.6.0 must be 
checked in by the b1 cutoff on Monday (unless you have contacted me and we have 
agreed on an extension).

As was done during the 3.5 release cycle, we will create the 3.6 branch at b1 
time.  During the beta phase, the emphasis is on fixes for new features, fixes 
for all categories of bugs and regressions, and documentation fixes/updates.  I 
will send out specific information for core committers next week after the 
creation of the b1 tag and the 3.6 branch.

Beta releases are intended to give the wider community the opportunity to test 
new features and bug fixes and to prepare their projects to support the new 
feature release.  We strongly encourage maintainers of third-party Python 
projects to test with 3.6 during the beta phase and report issues found to 
bugs.python.org as soon as possible.  While the release will be feature 
complete entering the beta phase, it is possible that features may be modified 
or, in rare cases, deleted up until the start of the release candidate phase.  
Our goal is have no changes after rc1.  To achieve that, it will be extremely 
important to get as much exposure for 3.6 as possible during the beta phase.

To recap:

2016-09-12 ~12:00 UTC: code snapshot for 3.6.0 beta 1 (feature code freeze, no 
new features)

2016-09-12 3.6 branch opens for 3.6.0; 3.7.0 feature development begins

2016-09-12 to 2016-12-04: 3.6.0 beta phase (bug, regression, and doc fixes, no 
new features)

2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze)

2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes)

2018-06 3.7.0 release (3.6.0 release + 18 months, details TBD)


Thank you all for your great efforts so far on 3.6; it should be a great 
release!

--Ned

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Chris Angelico]

On Fri, Sep 9, 2016 at 6:22 AM, Victor Stinner  wrote:
> A nice "side effect" of compact dict is that the dictionary now
> preserves the insertion order. It means that keyword arguments can now
> be iterated by their creation order:
>

This is pretty sweet! Of course, there are going to be 1172 complaints
from people who's doctests have been broken, same as when hash
randomization came in, but personally, I don't care. Thank you for
landing this!

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Review request: issue 27350, compact ordered dict

[Guido van Rossum]

It's in! Congrats, and thanks for your great work! See longer post by Victor.

On Sun, Aug 28, 2016 at 12:16 AM, INADA Naoki  wrote:
> On Sun, Aug 28, 2016 at 2:05 PM, Guido van Rossum  wrote:
>> Hopefully some core dev(s) can work on this during the core sprint, which is
>> from Sept 5-9.
>>
>
> OK. While I'm in Japan (UTC+9) and cannot join the sprint, I'll be
> active as possible
> while the sprint.
>
> Thank you!
>
>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>
> --
> INADA Naoki  



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Guido van Rossum]

On Thu, Sep 8, 2016 at 1:36 PM, Guido van Rossum  wrote:
> IIUC there's one small thing we might still want to change somewhere
> after 3.6b1 but before 3.6rc1: the order is not preserved when you
> delete some keys and then add some other keys. Apparently PyPy has
> come up with a clever solution for this, and we should probably adopt
> it, but it's probably best not to hurry that for 3.6b1.

It turns out I was mistaken. Naoki's implementation *does* preserve
order across deletions. So we are already up to the standard set by
PyPy. Go Naoki!!

PS. As a consequence we're also going to change 520. Sit tight!

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Victor Stinner]

2016-09-08 13:36 GMT-07:00 Guido van Rossum :
> IIUC there's one small thing we might still want to change somewhere
> after 3.6b1 but before 3.6rc1: the order is not preserved when you
> delete some keys and then add some other keys. Apparently PyPy has
> come up with a clever solution for this, and we should probably adopt
> it, but it's probably best not to hurry that for 3.6b1.

Very good news: I was wrong, Raymond Hettinger confirmed that the
Python 3.6 dict *already* preserves the items order in all cases. In
short, Python 3.6 dict = Python 3.5 OrderedDict (in fact, OrderedDict
has a few more methods).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Random832]

On Thu, Sep 8, 2016, at 16:01, Chris Barker wrote:
> Is there a "long" in there anywhere in the integer implementation?

The python 2 long type is the python 3 int type. The python 2 int type
is gone.

> I don't have py3 running on win64 anywhere right now, but in win64 py2,
> that would give you:
> 
> dtype('int32')
> 
> as it's a "long" under the hood

That's numpy's decision, there's nothing "built-in" about it.

> (and I'm pretty sure that is not because of numpy code itself, but rather
> how Cpython is written/compiled)

Nope.
https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/common.c#L105

Note that PyInt_Check doesn't exist anymore in Python 3. NumPy provides
its own definition:

https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/npy_3kcompat.h#L35
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Guido van Rossum]

Can you guys get a room? There is absolutely no reason that all of
python-dev needs to hear this.

On Thu, Sep 8, 2016 at 1:35 PM, Steve Dower  wrote:
> On 08Sep2016 1301, Chris Barker wrote:
>>
>> On Thu, Sep 8, 2016 at 9:39 AM, Random832 > > wrote:
>>
>> You're talking about changing Py_ssize_t, right?
>>
>>
>> wouldn't that be the pointer size?
>>
>> Is there a "long" in there anywhere in the integer implementation?
>> [SNIP]
>> Does py3 already use int64?
>
>
> Py3 has used a variable-length int representation for its entire existence.
>
> Python 3.0.1 (r301:69561, Feb 13 2009, 20:04:18) [MSC v.1500 32 bit (Intel)]
> on win32
> Type "help", "copyright", "credits" or "license" for more information.
 2**1000
> 10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376
>
> Cheers,
> Steve
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Guido van Rossum]

Thanks Victor for the review and commit, and thanks Naoki for your
truly amazing implementation work!!

I've also accepted Eric's PEP 468.

IIUC there's one small thing we might still want to change somewhere
after 3.6b1 but before 3.6rc1: the order is not preserved when you
delete some keys and then add some other keys. Apparently PyPy has
come up with a clever solution for this, and we should probably adopt
it, but it's probably best not to hurry that for 3.6b1.

--Guido

On Thu, Sep 8, 2016 at 1:22 PM, Victor Stinner  wrote:
> Hi,
>
> I pushed INADA Naoki's implementation of the "compact dict". The hash
> table now stores indices pointing to a new second table which contains
> keys and values: it adds one new level of indirection. The table of
> indices is "compact": use 1, 2, 4 or 8 bytes per indice depending on
> the size of the dictionary. Moreover, the keys/values table is also
> more compact: its size is 2/3 of the indices table.
>
> A nice "side effect" of compact dict is that the dictionary now
> preserves the insertion order. It means that keyword arguments can now
> be iterated by their creation order:
>
> Python 3.5.1 (default, Jun 20 2016, 14:48:22)
 def func(**kw): print(kw.keys())
> ...
 func(a=1, b=2, c=3, d=4, e=5)
> dict_keys(['c', 'd', 'e', 'b', 'a'])   # random order
>
> vs
>
> Python 3.6.0a4+ (default:d43f819caea7, Sep  8 2016, 13:05:34)
 def func(**kw): print(kw.keys())
> ...
 func(a=1, b=2, c=3, d=4, e=5)
> dict_keys(['a', 'b', 'c', 'd', 'e'])   # expected order
>
>
> It means that the main goal of the PEP 468 "Preserving the order of
> **kwargs in a function" is now implemented in Python 3.6:
> https://www.python.org/dev/peps/pep-0468/
>
> But Eric Snow still wants to rephrase the PEP 468 to replace
> "OrderedDict" with "ordered mapping".
>
>
> For more information on compact dict, see:
>
> * http://bugs.python.org/issue27350
> * https://mail.python.org/pipermail/python-dev/2016-June/145299.html
> * 
> https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html
> *https://mail.python.org/pipermail/python-dev/2012-December/123028.html
>
>
> PyPy also implements the "compact dict", but it uses further "tricks"
> to preserve the order even if items are removed and then others are
> added. We might also implement these tricks in CPython, so dict will
> be ordered as well!
>
> --
>
> Moreover, since Guido approved the PEP 509 "Add a private version to
> dict", I just pushed the implementation.
>
> The PEP 509 provides a C API (a dict version field) to implement
> efficient caches on namespaces. It might be used to implement a cache
> on builtins in Python 3.6 using Yury's opcode cache (stay tuned!).
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Steve Dower]

On 08Sep2016 1301, Chris Barker wrote:

On Thu, Sep 8, 2016 at 9:39 AM, Random832 > wrote:

You're talking about changing Py_ssize_t, right?


wouldn't that be the pointer size?

Is there a "long" in there anywhere in the integer implementation?
[SNIP]
Does py3 already use int64?


Py3 has used a variable-length int representation for its entire existence.

Python 3.0.1 (r301:69561, Feb 13 2009, 20:04:18) [MSC v.1500 32 bit 
(Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.
>>> 2**1000
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 468 ready for pronouncement.

[Guido van Rossum]

Thanks Eric!

The synergy between this PEP and the compact dict is amazing BTW.
Clearly its time has come. Therefore:

PEP 468 is now accepted. You may as well call it Final, since all we
need to do now is update the docs. Congrats!!

--Guido

On Thu, Sep 8, 2016 at 1:20 PM, Eric Snow  wrote:
> see: https://github.com/python/peps/blob/master/pep-0468.txt
>
> With the introduction of the compact dict implementation for CPython
> 3.6, PEP 468 becomes no more than a change to the language reference.
> I've adjusted the PEP to specify use of an ordered mapping rather than
> exactly OrderedDict.  Thanks!
>
> -eric
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Guido van Rossum]

Please no. Let's not add unrelated new functionality in with this
already large change with not entirely understood consequences.

On Thu, Sep 8, 2016 at 1:05 PM, Chris Barker  wrote:
> On Thu, Sep 8, 2016 at 10:35 AM, Random832  wrote:
>>
>>
>> It means that the so-called "bash" on windows 10 is actually a full
>> Ubuntu system (running on, AIUI, a simulation of Linux kernel system
>> calls), which will presumably also have its own python installation and
>> use a UTF-8 locale, rather than one that runs "natively" on win32.
>
>
> yes -- it looks like one could run a "linux" build of python under the whole
> subsystem, which would presumably "look" jsu tlike LInux to Python.
>
>
>>
>> If it's possible for a win32 version of python to call it as a
>> subprocess,
>
>
> But this is what I was referring too -- it may be way to early to know what
> the capabilities or implications are, but I'm hoping that "regular" windows
> programs can interact with the subsystem. So if we're making changes now, it
> would be nice to consider it if we can.
>
>>
>> Incidentally, according to
>>
>> https://github.com/Microsoft/BashOnWindows/issues/2, pipes didn't work
>> at all between WSL processes and Win32 processes until two weeks ago, so
>> it's clear that these features are still evolving.
>
>
> so it may indeed be way to early -- but if they DO work now -- pretty cool!
>
> Thanks,
>
>-CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 3.6 dict becomes compact and gets a private version; and keywords become ordered

[Victor Stinner]

Hi,

I pushed INADA Naoki's implementation of the "compact dict". The hash
table now stores indices pointing to a new second table which contains
keys and values: it adds one new level of indirection. The table of
indices is "compact": use 1, 2, 4 or 8 bytes per indice depending on
the size of the dictionary. Moreover, the keys/values table is also
more compact: its size is 2/3 of the indices table.

A nice "side effect" of compact dict is that the dictionary now
preserves the insertion order. It means that keyword arguments can now
be iterated by their creation order:

Python 3.5.1 (default, Jun 20 2016, 14:48:22)
>>> def func(**kw): print(kw.keys())
...
>>> func(a=1, b=2, c=3, d=4, e=5)
dict_keys(['c', 'd', 'e', 'b', 'a'])   # random order

vs

Python 3.6.0a4+ (default:d43f819caea7, Sep  8 2016, 13:05:34)
>>> def func(**kw): print(kw.keys())
...
>>> func(a=1, b=2, c=3, d=4, e=5)
dict_keys(['a', 'b', 'c', 'd', 'e'])   # expected order


It means that the main goal of the PEP 468 "Preserving the order of
**kwargs in a function" is now implemented in Python 3.6:
https://www.python.org/dev/peps/pep-0468/

But Eric Snow still wants to rephrase the PEP 468 to replace
"OrderedDict" with "ordered mapping".


For more information on compact dict, see:

* http://bugs.python.org/issue27350
* https://mail.python.org/pipermail/python-dev/2016-June/145299.html
* 
https://morepypy.blogspot.jp/2015/01/faster-more-memory-efficient-and-more.html
*https://mail.python.org/pipermail/python-dev/2012-December/123028.html


PyPy also implements the "compact dict", but it uses further "tricks"
to preserve the order even if items are removed and then others are
added. We might also implement these tricks in CPython, so dict will
be ordered as well!

--

Moreover, since Guido approved the PEP 509 "Add a private version to
dict", I just pushed the implementation.

The PEP 509 provides a C API (a dict version field) to implement
efficient caches on namespaces. It might be used to implement a cache
on builtins in Python 3.6 using Yury's opcode cache (stay tuned!).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 468 ready for pronouncement.

[Eric Snow]

see: https://github.com/python/peps/blob/master/pep-0468.txt

With the introduction of the compact dict implementation for CPython
3.6, PEP 468 becomes no more than a change to the language reference.
I've adjusted the PEP to specify use of an ordered mapping rather than
exactly OrderedDict.  Thanks!

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Chris Barker]

On Thu, Sep 8, 2016 at 1:14 PM, Guido van Rossum  wrote:

> Please no. Let's not add unrelated new functionality in with this
> already large change with not entirely understood consequences.
>

Fair enough -- this is clearly a really raw API so far.

-CHB





>
> On Thu, Sep 8, 2016 at 1:05 PM, Chris Barker 
> wrote:
> > On Thu, Sep 8, 2016 at 10:35 AM, Random832 
> wrote:
> >>
> >>
> >> It means that the so-called "bash" on windows 10 is actually a full
> >> Ubuntu system (running on, AIUI, a simulation of Linux kernel system
> >> calls), which will presumably also have its own python installation and
> >> use a UTF-8 locale, rather than one that runs "natively" on win32.
> >
> >
> > yes -- it looks like one could run a "linux" build of python under the
> whole
> > subsystem, which would presumably "look" jsu tlike LInux to Python.
> >
> >
> >>
> >> If it's possible for a win32 version of python to call it as a
> >> subprocess,
> >
> >
> > But this is what I was referring too -- it may be way to early to know
> what
> > the capabilities or implications are, but I'm hoping that "regular"
> windows
> > programs can interact with the subsystem. So if we're making changes
> now, it
> > would be nice to consider it if we can.
> >
> >>
> >> Incidentally, according to
> >>
> >> https://github.com/Microsoft/BashOnWindows/issues/2, pipes didn't work
> >> at all between WSL processes and Win32 processes until two weeks ago, so
> >> it's clear that these features are still evolving.
> >
> >
> > so it may indeed be way to early -- but if they DO work now -- pretty
> cool!
> >
> > Thanks,
> >
> >-CHB
> >
> >
> > --
> >
> > Christopher Barker, Ph.D.
> > Oceanographer
> >
> > Emergency Response Division
> > NOAA/NOS/OR(206) 526-6959   voice
> > 7600 Sand Point Way NE   (206) 526-6329   fax
> > Seattle, WA  98115   (206) 526-6317   main reception
> >
> > chris.bar...@noaa.gov
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Chris Barker]

On Thu, Sep 8, 2016 at 10:35 AM, Random832  wrote:

>
> It means that the so-called "bash" on windows 10 is actually a full
> Ubuntu system (running on, AIUI, a simulation of Linux kernel system
> calls), which will presumably also have its own python installation and
> use a UTF-8 locale, rather than one that runs "natively" on win32.
>

yes -- it looks like one could run a "linux" build of python under the
whole subsystem, which would presumably "look" jsu tlike LInux to Python.



> If it's possible for a win32 version of python to call it as a
> subprocess,


But this is what I was referring too -- it may be way to early to know what
the capabilities or implications are, but I'm hoping that "regular" windows
programs can interact with the subsystem. So if we're making changes now,
it would be nice to consider it if we can.


> Incidentally, according to
>
https://github.com/Microsoft/BashOnWindows/issues/2, pipes didn't work
> at all between WSL processes and Win32 processes until two weeks ago, so
> it's clear that these features are still evolving.


so it may indeed be way to early -- but if they DO work now -- pretty cool!

Thanks,

   -CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Chris Barker]

On Thu, Sep 8, 2016 at 9:39 AM, Random832  wrote:

> You're talking about changing Py_ssize_t, right?
>

wouldn't that be the pointer size?

Is there a "long" in there anywhere in the integer implementation?

My example is this:

on OS-X, py3.5:

import numpy as np

In [9]: arr = np.array([1,2,3])

Out[10]: array([1, 2, 3])

In [11]: arr.dtype

Out[11]: dtype('int64')

I don't have py3 running on win64 anywhere right now, but in win64 py2,
that would give you:

dtype('int32')

as it's a "long" under the hood

(and I'm pretty sure that is not because of numpy code itself, but rather
how Cpython is written/compiled)

Does py3 already use int64?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Random832]

On Thu, Sep 8, 2016, at 13:10, Guido van Rossum wrote:
> On Thu, Sep 8, 2016 at 9:57 AM, Brett Cannon  wrote:
> > Bash on Windows is just Linux, so it isn't affected by any of this.
> 
> I don't know what that sentence means.

It means that the so-called "bash" on windows 10 is actually a full
Ubuntu system (running on, AIUI, a simulation of Linux kernel system
calls), which will presumably also have its own python installation and
use a UTF-8 locale, rather than one that runs "natively" on win32.

If it's possible for a win32 version of python to call it as a
subprocess, this may be an argument in favor of using UTF-8 - subject to
finding out whether WSL does use UTF-8, whether it supports non-ASCII
arguments from a Win32 CreateProcess at all, whether there's any way to
pass non-UTF-8 arguments to it, etc.

Incidentally, according to
https://github.com/Microsoft/BashOnWindows/issues/2, pipes didn't work
at all between WSL processes and Win32 processes until two weeks ago, so
it's clear that these features are still evolving.

> But anyways, if someone wants
> to try making subprocess work with bytes arguments on Windows work,
> that's just a bugfix, and you're not constrained by how it works on
> previous Python versions (since it doesn't work there at all). It
> might be wise to choose an interpretation that's consistent with other
> uses of command line arguments by Python on Windows though (rather
> than choosing to favor making just bash work the same as it works on
> Linux).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Guido van Rossum]

On Thu, Sep 8, 2016 at 9:57 AM, Brett Cannon  wrote:
>
>
> On Thu, 8 Sep 2016 at 09:06 Chris Barker  wrote:
>>
>> On Wed, Sep 7, 2016 at 10:37 AM, Guido van Rossum 
>> wrote:
>>>
>>> And apart from Python, few shell commands that work on
>>> Unix make much sense on Windows,
>>
>>
>> Does the (optional) addition of bash to Windows 10 have any impact on
>> this?
>>
>> It'll be something that Windows developers can't count on their users
>> having for a good while, if ever, but if you can control the deployment
>> environment, then you might. And it would be VERY tempting for
>> "posix-focused" developers that want to run their code on Windows.
>>
>> So it would be nice if the "new" approach worked well with bash on
>> Windows.
>
>
> Bash on Windows is just Linux, so it isn't affected by any of this.

I don't know what that sentence means. But anyways, if someone wants
to try making subprocess work with bytes arguments on Windows work,
that's just a bugfix, and you're not constrained by how it works on
previous Python versions (since it doesn't work there at all). It
might be wise to choose an interpretation that's consistent with other
uses of command line arguments by Python on Windows though (rather
than choosing to favor making just bash work the same as it works on
Linux).


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Chris Angelico]

On Fri, Sep 9, 2016 at 2:39 AM, Random832  wrote:
> On Thu, Sep 8, 2016, at 12:30, Chris Barker wrote:
>> That's why I said "based on" -- under the hood, a C type is used, and
>> IIUC, that type has been "long" for ages. And a long on Windows 64
>> (with the MS compiler anyway) is 32 bit, and a long on *nix (with the
>> gnu compilers, at least) is 64 bits.
>>
>> This doesn't expose itself to pure python (and sys.maxint is now gone)
>> but it does get exposed in the C API, and in particular, when passing
>> data back and forth between numpy and pure python (numpy doesn't
>> support an unlimited integer like python), or working with buffers or
>> bytearrays, or whatever in Cython.
>
> I'm not sure "the builtin integer type" was the right term for what
> you're referring to.
>
> You're talking about changing Py_ssize_t, right?

There are a few places where the size of ssize_t becomes visible to a
Python script.

Python 3.6.0a4+ (default:4b64a049f451+, Aug 19 2016, 23:41:43)
[GCC 6.1.1 20160802] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x=1<<(1<<30)
>>> x=1<<(1<<34)
>>> x=1<<(1<<62)
Traceback (most recent call last):
  File "", line 1, in 
MemoryError
>>> x=1<<(1<<66)
Traceback (most recent call last):
  File "", line 1, in 
OverflowError: Python int too large to convert to C ssize_t

But I got the same result on 3.5.2 on Win 7 64-bit, so I'm not seeing
a difference here - it seems that PyLong_AsSsize_t has the same limits
on both platforms.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Brett Cannon]

On Thu, 8 Sep 2016 at 09:06 Chris Barker  wrote:

> On Wed, Sep 7, 2016 at 10:37 AM, Guido van Rossum 
> wrote:
>
>> And apart from Python, few shell commands that work on
>> Unix make much sense on Windows,
>
>
> Does the (optional) addition of bash to Windows 10 have any impact on this?
>
> It'll be something that Windows developers can't count on their users
> having for a good while, if ever, but if you can control the deployment
> environment, then you might. And it would be VERY tempting for
> "posix-focused" developers that want to run their code on Windows.
>
> So it would be nice if the "new" approach worked well with bash on Windows.
>

Bash on Windows is just Linux, so it isn't affected by any of this.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Random832]

On Thu, Sep 8, 2016, at 12:30, Chris Barker wrote:
> That's why I said "based on" -- under the hood, a C type is used, and
> IIUC, that type has been "long" for ages. And a long on Windows 64
> (with the MS compiler anyway) is 32 bit, and a long on *nix (with the
> gnu compilers, at least) is 64 bits.
>
> This doesn't expose itself to pure python (and sys.maxint is now gone)
> but it does get exposed in the C API, and in particular, when passing
> data back and forth between numpy and pure python (numpy doesn't
> support an unlimited integer like python), or working with buffers or
> bytearrays, or whatever in Cython.

I'm not sure "the builtin integer type" was the right term for what
you're referring to.

You're talking about changing Py_ssize_t, right?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Chris Barker]

On Thu, Sep 8, 2016 at 9:17 AM, Benjamin Peterson 
wrote:

> > Does this mean that we might be able to have the built-in integer be
> > based
> > on int64_t now? so Windows64 and *nix64 will be the same?
>
> The builtin integer type (in Python 3) is variable length.
>

indeed it is -- py2.7 also??

That's why I said "based on" -- under the hood, a C type is used, and IIUC,
that type has been "long" for ages. And a long on Windows 64 (with the MS
 compiler anyway) is 32 bit, and a long on *nix (with the gnu compilers, at
least) is 64 bits.

This doesn't expose itself to pure python (and sys.maxint is now gone) but
it does get exposed in the C API, and in particular, when passing data back
and forth between numpy and pure python (numpy doesn't support an unlimited
integer like python), or working with buffers or bytearrays, or whatever in
Cython.

Perhaps this is now a non-issue in py3 -- I honestly have not done any
"real" computation work with py3 yet, but it sure is in 2.7

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Benjamin Peterson]

On Thu, Sep 8, 2016, at 09:09, Chris Barker wrote:
> > > - Standard integer types in  and 
> >
> 
> 
> > Yes, I will clarify we require the fixed-width types.
> 
> 
> Does this mean that we might be able to have the built-in integer be
> based
> on int64_t now? so Windows64 and *nix64 will be the same?

The builtin integer type (in Python 3) is variable length.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (some) C99 added to PEP 7

[Chris Barker]

> > - Standard integer types in  and 
>


> Yes, I will clarify we require the fixed-width types.


Does this mean that we might be able to have the built-in integer be based
on int64_t now? so Windows64 and *nix64 will be the same?

- CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Chris Barker]

On Wed, Sep 7, 2016 at 10:37 AM, Guido van Rossum  wrote:

> And apart from Python, few shell commands that work on
> Unix make much sense on Windows,


Does the (optional) addition of bash to Windows 10 have any impact on this?

It'll be something that Windows developers can't count on their users
having for a good while, if ever, but if you can control the deployment
environment, then you might. And it would be VERY tempting for
"posix-focused" developers that want to run their code on Windows.

So it would be nice if the "new" approach worked well with bash on Windows.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] hg push segfault

[Benjamin Peterson]

On Thu, Sep 8, 2016, at 02:10, Christian Heimes wrote:
> Hi,
> 
> About 10 minutes ago I got a couple of remote segfaults from
> hg.python.org. They occurred during push and pull operations:
> 
> $ hg push
> pushing to ssh://h...@hg.python.org/cpython
> remote: bash: line 1: 25019 Segmentation fault
> HGPUSHER=christian.heimes /srv/hg/bin/hg-ssh /srv/hg/repos/*
> abort: no suitable response from remote hg!
> 
> It's fine again now. Can somebody look into the matter, please?

A little bit after this the OOM killer started running, so it's probably
related.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.5): supress coroutine warning when an exception is pending (#27968)

[Benjamin Peterson]

On Thu, Sep 8, 2016, at 04:09, Christian Heimes wrote:
> On 2016-09-07 17:47, benjamin.peterson wrote:
> > https://hg.python.org/cpython/rev/234f758449f8
> > changeset:   103223:234f758449f8
> > branch:  3.5
> > parent:  103213:7537ca1c2aaf
> > user:Benjamin Peterson 
> > date:Wed Sep 07 08:46:59 2016 -0700
> > summary:
> >   supress coroutine warning when an exception is pending (#27968)
> > 
> > files:
> >   Objects/genobject.c |  27 +++
> >   1 files changed, 15 insertions(+), 12 deletions(-)
> > 
> > 
> > diff --git a/Objects/genobject.c b/Objects/genobject.c
> > --- a/Objects/genobject.c
> > +++ b/Objects/genobject.c
> > @@ -21,7 +21,7 @@
> >  _PyGen_Finalize(PyObject *self)
> >  {
> >  PyGenObject *gen = (PyGenObject *)self;
> > -PyObject *res;
> > +PyObject *res = NULL;
> >  PyObject *error_type, *error_value, *error_traceback;
> >  
> >  if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL)
> > @@ -33,23 +33,26 @@
> >  
> >  /* If `gen` is a coroutine, and if it was never awaited on,
> > issue a RuntimeWarning. */
> > -if (gen->gi_code != NULL
> > -&& ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE
> > -&& gen->gi_frame->f_lasti == -1
> > -&& !PyErr_Occurred()
> > -&& PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
> > -"coroutine '%.50S' was never awaited",
> > -gen->gi_qualname)) {
> > -res = NULL;  /* oops, exception */
> > +if (gen->gi_code != NULL &&
> > +((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE &&
> > +gen->gi_frame->f_lasti == -1) {
> > +if (!error_value) {
> > +PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
> > + "coroutine '%.50S' was never awaited",
> > + gen->gi_qualname);
> > +}
> 
> You don't check the return value of PyErr_WarnFormat(). It does not
> signal an exception in case warnings are turned into exceptions.

It's checked by PyErr_Occurred() several lines later.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.5): supress coroutine warning when an exception is pending (#27968)

[Christian Heimes]

On 2016-09-07 17:47, benjamin.peterson wrote:
> https://hg.python.org/cpython/rev/234f758449f8
> changeset:   103223:234f758449f8
> branch:  3.5
> parent:  103213:7537ca1c2aaf
> user:Benjamin Peterson 
> date:Wed Sep 07 08:46:59 2016 -0700
> summary:
>   supress coroutine warning when an exception is pending (#27968)
> 
> files:
>   Objects/genobject.c |  27 +++
>   1 files changed, 15 insertions(+), 12 deletions(-)
> 
> 
> diff --git a/Objects/genobject.c b/Objects/genobject.c
> --- a/Objects/genobject.c
> +++ b/Objects/genobject.c
> @@ -21,7 +21,7 @@
>  _PyGen_Finalize(PyObject *self)
>  {
>  PyGenObject *gen = (PyGenObject *)self;
> -PyObject *res;
> +PyObject *res = NULL;
>  PyObject *error_type, *error_value, *error_traceback;
>  
>  if (gen->gi_frame == NULL || gen->gi_frame->f_stacktop == NULL)
> @@ -33,23 +33,26 @@
>  
>  /* If `gen` is a coroutine, and if it was never awaited on,
> issue a RuntimeWarning. */
> -if (gen->gi_code != NULL
> -&& ((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE
> -&& gen->gi_frame->f_lasti == -1
> -&& !PyErr_Occurred()
> -&& PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
> -"coroutine '%.50S' was never awaited",
> -gen->gi_qualname)) {
> -res = NULL;  /* oops, exception */
> +if (gen->gi_code != NULL &&
> +((PyCodeObject *)gen->gi_code)->co_flags & CO_COROUTINE &&
> +gen->gi_frame->f_lasti == -1) {
> +if (!error_value) {
> +PyErr_WarnFormat(PyExc_RuntimeWarning, 1,
> + "coroutine '%.50S' was never awaited",
> + gen->gi_qualname);
> +}

You don't check the return value of PyErr_WarnFormat(). It does not
signal an exception in case warnings are turned into exceptions.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] hg push segfault

[Christian Heimes]

Hi,

About 10 minutes ago I got a couple of remote segfaults from
hg.python.org. They occurred during push and pull operations:

$ hg push
pushing to ssh://h...@hg.python.org/cpython
remote: bash: line 1: 25019 Segmentation fault
HGPUSHER=christian.heimes /srv/hg/bin/hg-ssh /srv/hg/repos/*
abort: no suitable response from remote hg!

It's fine again now. Can somebody look into the matter, please?

Christian

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com