[Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython)

2018-07-25 Thread Erik Bray 
On Wed, Nov 8, 2017 at 3:39 PM Erik Bray  wrote:
>
> Hi folks,
>
> As some people here know I've been working off and on for a while to
> improve CPython's support of Cygwin.  I'm motivated in part by a need
> to have software working on Python 3.x on Cygwin for the foreseeable
> future, preferably with minimal graft.  (As an incidental side-effect
> Python's test suite--especially of system-level functionality--serves
> as an interesting test suite for Cygwin itself too.)
>
> This is partly what motivated PEP 539 [1], although that PEP had the
> advantage of benefiting other POSIX-compatible platforms as well (and
> in fact was fixing an aspect of CPython that made it unfriendly to
> supporting other platforms).
>
> As far as I can tell, the first commit to Python to add any kind of
> support for Cygwin was made by Guido (committing a contributed patch)
> back in 1999 [2].  Since then, bits and pieces have been added for
> Cygwin's benefit over time, with varying degrees of impact in terms of
> #ifdefs and the like (for the most part Cygwin does not require *much*
> in the way of special support, but it does have some differences from
> a "normal" POSIX-compliant platform, such as the possibility for
> case-insensitive filesystems and executables that end in .exe).  I
> don't know whether it's ever been "officially supported" but someone
> with a longer memory of the project can comment on that.  I'm not sure
> if it was discussed at all or not in the context of PEP 11.
>
> I have personally put in a fair amount of effort already in either
> fixing issues on Cygwin (many of these issues also impact MinGW), or
> more often than not fixing issues in the CPython test suite on
> Cygwin--these are mostly tests that are broken due to invalid
> assumptions about the platform (for example, that there is always a
> "root" user with uid=0; this is not the case on Cygwin).  In other
> cases some tests need to be skipped or worked around due to
> platform-specific bugs, and Cygwin is hardly the only case of this in
> the test suite.
>
> I also have an experimental AppVeyor configuration for running the
> tests on Cygwin [3], as well as an experimental buildbot (not
> available on the internet, but working).  These currently rely on a
> custom branch that includes fixes needed for the test suite to run to
> completion without crashing or hanging (e.g.
> https://bugs.python.org/issue31885).  It would be nice to add this as
> an official buildbot, but I'm not sure if it makes sense to do that
> until it's "green", or at least not crashing.  I have several other
> patches to the tests toward this goal, and am currently down to ~22
> tests failing.
>
> Before I do any more work on this, however, it would be best to once
> and for all clarify the support for Cygwin in CPython, as it has never
> been "officially supported" nor unsupported--this way we can avoid
> having this discussion every time a patch related to Cygwin comes up.
> I could provide some arguments for why I believe Cygwin should
> supported, but before this gets too long I'd just like to float the
> idea of having the discussion in the first place.  It's also not
> exactly clear to me how to meet the standards in PEP 11 for supporting
> a platform--in particular it's not clear when a buildbot is considered
> "stable", or how to achieve that without getting necessary fixes
> merged into the main branch in the first place.
>
> Thanks,
> Erik
>
>
>
> [1] https://www.python.org/dev/peps/pep-0539/
> [2] 
> https://github.com/python/cpython/commit/717d1fdf2acbef5e6b47d9b4dcf48ef1829be685
> [3] https://ci.appveyor.com/project/embray/cpython

Apologies for responding to a months old post, but rather than repeat
myself verbatim I'll just mention that all of the above is still true
and relevant, and I am still interested in getting Python somewhere
closer to "stable" on Cygwin.

Part of the problem with my previous approach is that I was trying to
fix every last test failure before asking to add Cygwin to CPython's
CI fleet.  While I believe all failures *should* be fixed (or skipped
as appropriate) this is not practical to do in a short amount of time,
and not having CI implemented for a platform means new bugs are added
faster than we can fix the existing bugs.  For example, between 3.6
and 3.7 two new bugs have caused Python to be unbuildable on Cygwin:

https://bugs.python.org/issue34211
https://bugs.python.org/issue34212

This is in addition to an older issue that I was hoping to have fixed
in Python 3.7, as the PR was "green" for some time well before its
release:

https://github.com/python/cpython/pull/4348

However, I fear part of why it was never merged is that lack of CI for
Cygwin, which is in turn because it was impossible to get a "stable"
test pass on Cygwin in the first place--something of a catch-22.

I think a new approach that might be more practical for actually
getting this platform re-supported, is to go ahead and add a CI build,
and just skip al

[Python-Dev] Tests failing on Windows with TESTFN

2018-07-25 Thread Tim Golden 
I'm just easing back into core development work by trying to get a 
stable testing environment for Python development on Windows.


One problem is that certain tests use support.TESTFN (a local directory 
constructed from the pid) for output files etc. However this can cause 
issues on Windows when recreating the folders / files for multiple 
tests, especially when running in parallel.


Here's an example on my laptop deliberately running 3 tests with -j0 
which I know will generate an error about one time in three:


C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 
test_importlib


Running Debug|Win32 interpreter...
Run tests in parallel using 6 child processes
0:00:23 [1/3/1] test_urllib2 failed
test test_urllib2 failed -- Traceback (most recent call last):
  File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line 
821, in test_file

f = open(TESTFN, "wb")
PermissionError: [Errno 13] Permission denied: '@test_15564_tmp'

Although these errors are both intermittent and fairly easily spotted, 
the effect is that I rarely get a clean test run when I'm applying a patch.


I started to address this years ago but things stalled. I'm happy to 
pick this up again and have another go, but I wanted to ask first 
whether there was any objection to my converting tests to using tempfile 
functions which should avoid the problem?


TJG
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Tests failing on Windows with TESTFN

2018-07-25 Thread Tim Golden 

[trying again; sorry if it shows up twice]

I'm just easing back into core development work by trying to get a 
stable testing environment for Python development on Windows.


One problem is that certain tests use support.TESTFN (a local directory 
constructed from the pid) for output files etc. However this can cause 
issues on Windows when recreating the folders / files for multiple 
tests, especially when running in parallel.


Here's an example on my laptop deliberately running 3 tests with -j0 
which I know will generate an error about one time in three:


C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 
test_importlib


Running Debug|Win32 interpreter...
Run tests in parallel using 6 child processes
0:00:23 [1/3/1] test_urllib2 failed
test test_urllib2 failed -- Traceback (most recent call last):
  File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line 
821, in test_file

f = open(TESTFN, "wb")
PermissionError: [Errno 13] Permission denied: '@test_15564_tmp'

Although these errors are both intermittent and fairly easily spotted, 
the effect is that I rarely get a clean test run when I'm applying a patch.


I started to address this years ago but things stalled. I'm happy to 
pick this up again and have another go, but I wanted to ask first 
whether there was any objection to my converting tests to using tempfile 
functions which should avoid the problem?


TJG
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-25 Thread Erik Bray 
On Sat, Jul 21, 2018 at 6:30 PM Jeroen Demeyer  wrote:
>
> Hello,
>
> I finally managed to get some real-life benchmarks for why we need a
> faster C calling protocol (see PEPs 576, 579, 580).
>
> I focused on the Cython compilation of SageMath. By default, a function
> in Cython is an instance of builtin_function_or_method (analogously,
> method_descriptor for a method), which has special optimizations in the
> CPython interpreter. But the option "binding=True" changes those to a
> custom class which is NOT optimized.
>
> I ran the full SageMath testsuite several times without and with
> binding=True to find out any significant differences. The most dramatic
> difference is multiplication for generic matrices. More precisely, with
> the following command:
>
> python -m timeit -s "from sage.all import MatrixSpace, GF; M =
> MatrixSpace(GF(9), 200).random_element()" "M * M"
>
> With binding=False, I got
> 10 loops, best of 3: 692 msec per loop
>
> With binding=True, I got
> 10 loops, best of 3: 1.16 sec per loop
>
> This is a big regression which should be gone completely with PEP 580.
>
> I should mention that this was done on Python 2.7.15 (SageMath is not
> yet ported to Python 3) but I see no reason why the conclusions
> shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1
> and Cython 0.28.4.

I haven't fully caught up on the thread yet so this might already be a
moot point.  But just in case it isn't, the Python 3 port of Sage
works well enough (at least on my branch) that the above benchmark
works, and would probably be worth repeating there (it's currently
Python 3.6.1, but upgrading to 3.7 probably wouldn't break the example
benchmark either).
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Fw:[issue34221] Any plans to combine collections.OrderedDict with dict

2018-07-25 Thread Zhao Lee 


Since Python 3.7，dicts remember the order that items were inserted, so any 
plans to combine collections.OrderedDict with dict?
https://docs.python.org/3/library/collections.html?#collections.OrderedDict
https://docs.python.org/3/library/stdtypes.html#dict

BTW, I think it would be better to move "Dictionaries preserve insertion order" 
part at the end of the doc of 
https://docs.python.org/3/library/stdtypes.html#dict
up below the doc for
class dict(**kwarg)
class dict(mapping, **kwarg)
class dict(iterable, **kwarg)
so that people won't miss the feature when consulting the doc



___
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fw:[issue34221] Any plans to combine collections.OrderedDict with dict

2018-07-25 Thread INADA Naoki 
On Thu, Jul 26, 2018 at 12:04 PM Zhao Lee  wrote:
>
>
> Since Python 3.7，dicts remember the order that items were inserted, so any 
> plans to combine collections.OrderedDict with dict?
> https://docs.python.org/3/library/collections.html?#collections.OrderedDict
> https://docs.python.org/3/library/stdtypes.html#dict

No.  There are some major difference.

* d1 == d2 ignores order / od1 == od2 compares order
* OrderedDict has move_to_end() method.
* OrderedDict.pop() takes `last=True` keyword.

Regards,
-- 
INADA Naoki  
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [issue34221] Any plans to combine collections.OrderedDict with dict

2018-07-25 Thread Raymond Hettinger 


> On Jul 25, 2018, at 8:23 PM, INADA Naoki  wrote:
> 
> On Thu, Jul 26, 2018 at 12:04 PM Zhao Lee  wrote:
>> 
>> 
>> Since Python 3.7，dicts remember the order that items were inserted, so any 
>> plans to combine collections.OrderedDict with dict?
>> https://docs.python.org/3/library/collections.html?#collections.OrderedDict
>> https://docs.python.org/3/library/stdtypes.html#dict
> 
> No.  There are some major difference.
> 
> * d1 == d2 ignores order / od1 == od2 compares order
> * OrderedDict has move_to_end() method.
> * OrderedDict.pop() takes `last=True` keyword.

In addition to the API differences noted by Naoki, there are also 
implementation differences.  The regular dict implements a low-cost solution 
for common cases.  The OrderedDict has a more complex scheme that can handle 
frequent rearrangements (move_to_end operations) without touching, resizing, or 
reordering the underlying dictionary. Roughly speaking, regular dicts emphasize 
fast, space-efficient core dictionary operations over ordering requirements 
while OrderedDicts prioritize ordering operations over other considerations.

That said, now that regular dicts are ordered by default, the need for 
collections.OrderedDict() should diminish quite a bit.  Mostly, I think people 
will ignore OrderedDict unless their application heavily exercises move to end 
operations.


Raymond
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com