[Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Victor Stinner 
Hi,

Our buildbots are now even more stable than in my previous buildbot report.

Many random failures have been fixed, even if there are still many
rare random failures (most of them are related to multiprocessing).

Search for issues created by "haypo" (me!) with the Component "Tests"
to see all remaining issues. I'm trying to open an issue for *each*
buildbot failure! So yeah, we have a long list of 44 open issues!

Correct me if I'm wrong, but, for the first time, *all reference
leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master),
on *Linux and Windows*! Before, we mostly focused on the master branch
(called "default" in Mercurial) on Linux.

I also started to fix a few "memory block" leaks, most (or all?) of
them should also be fixed (on all branches, on Linux and Windows).

A new policy has been decided (on python-committers): if a commit
breaks too many buildbots and cannot fixed easily nor quickly, the
commit will be reverted, just to keep our buildbots "green". It
doesn't mean that the commit will be rejected forever. It's a
practical solution to give more time to write a proper fix, take time
to review it, and not have to be stressed to fix buildbots "ASAP".
Moreover, keeping green buildbots all the time allows to catch
regressions more quickly, which ease debug in many ways.

You have be warned! Now I will not hesitate to revert your change if
you break my little buildbots ;-)

I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT).
Breaking other platforms is less important, since other platforms
already have issues, but not enough developers interested to fix them.
Obviously, I'm interested in keeping more buildbots green, if someone
shows up and help me to fix remaining issues!

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Terry Reedy 

On 6/29/2017 11:09 AM, Victor Stinner wrote:

Hi,

Our buildbots are now even more stable than in my previous buildbot report.

Many random failures have been fixed, even if there are still many
rare random failures (most of them are related to multiprocessing).

Search for issues created by "haypo" (me!) with the Component "Tests"
to see all remaining issues. I'm trying to open an issue for *each*
buildbot failure! So yeah, we have a long list of 44 open issues!

Correct me if I'm wrong, but, for the first time, *all reference
leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master),
on *Linux and Windows*! Before, we mostly focused on the master branch
(called "default" in Mercurial) on Linux.

I also started to fix a few "memory block" leaks, most (or all?) of
them should also be fixed (on all branches, on Linux and Windows).

A new policy has been decided (on python-committers): if a commit
breaks too many buildbots and cannot fixed easily nor quickly, the
commit will be reverted, just to keep our buildbots "green". It
doesn't mean that the commit will be rejected forever. It's a
practical solution to give more time to write a proper fix, take time
to review it, and not have to be stressed to fix buildbots "ASAP".
Moreover, keeping green buildbots all the time allows to catch
regressions more quickly, which ease debug in many ways.

You have be warned! Now I will not hesitate to revert your change if
you break my little buildbots ;-)

I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT).
Breaking other platforms is less important, since other platforms
already have issues, but not enough developers interested to fix them.
Obviously, I'm interested in keeping more buildbots green, if someone
shows up and help me to fix remaining issues!


Great work.

How about compiler warnings (and errors)?  When I compile on Windows, 
there are a boatload of orange-yellow warnings.  Some are about using a 
deprecated featured; some about dodgy casts; some (I presume) about 
other things.  Should 'no warnings' be a goal?


I believe you once fixed some, but new commits added more.  Could 'no 
additional warnings' be a CI requirement for merging? I expect that a 
new "How to avoid compiler warnings on Windows" section of the devguide 
would be needed.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Steve Dower 

On 29Jun2017 1013, Terry Reedy wrote:

Great work.


Agreed. Thanks, Victor!

How about compiler warnings (and errors)?  When I compile on Windows, 
there are a boatload of orange-yellow warnings.  Some are about using a 
deprecated featured; some about dodgy casts; some (I presume) about 
other things.  Should 'no warnings' be a goal?


Yes, I think that's a good goal. We're quickly getting there as well - I 
just merged two contributions yesterday that should significantly reduce 
the number of warnings.


I believe you once fixed some, but new commits added more.  Could 'no 
additional warnings' be a CI requirement for merging? I expect that a 
new "How to avoid compiler warnings on Windows" section of the devguide 
would be needed.


This is probably not feasible without a *really good* section in the 
devguide. I would rather have warnings in the output than global 
suppressions, and suppressing warnings globally is the usual instinct. 
Locally suppressing warnings can be fairly hideous and is usually not 
portable.


Some warnings are also complicated because of the nature of CPython. For 
example, the socket module exposes deprecated CRT functions (on Windows) 
directly because the API of the socket module promises to provide the 
function directly. Changing to the safer function would break the API 
guarantee (except sometimes it won't... hence "complicated").


Noting in PR builds that there are new warnings would be great if 
possible. I'd be concerned about it becoming a hard requirement though - 
I much prefer to leave the final decision in the hands of trusted people 
and provide them enough information to make a good decision.


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Terry Reedy 

On 6/29/2017 1:54 PM, Steve Dower wrote:

On 29Jun2017 1013, Terry Reedy wrote:

Great work.


Agreed. Thanks, Victor!

How about compiler warnings (and errors)?  When I compile on Windows, 
there are a boatload of orange-yellow warnings.  Some are about using 
a deprecated featured; some about dodgy casts; some (I presume) about 
other things.  Should 'no warnings' be a goal?


Yes, I think that's a good goal. We're quickly getting there as well - I 
just merged two contributions yesterday that should significantly reduce 
the number of warnings.


Great.  I recompiled changes for the last day or two (about 100 c files) 
and see none. _socket aside, I presume some of those files were the ones 
you fixed.


I believe you once fixed some, but new commits added more.  Could 'no 
additional warnings' be a CI requirement for merging? I expect that a 
new "How to avoid compiler warnings on Windows" section of the 
devguide would be needed.


This is probably not feasible without a *really good* section in the 
devguide. I would rather have warnings in the output than global 
suppressions, and suppressing warnings globally is the usual instinct. 
Locally suppressing warnings can be fairly hideous and is usually not 
portable.


Some warnings are also complicated because of the nature of CPython. For 
example, the socket module exposes deprecated CRT functions (on Windows) 
directly because the API of the socket module promises to provide the 
function directly. Changing to the safer function would break the API 
guarantee (except sometimes it won't... hence "complicated").


Thanks for the explanation.  Does 'deprecated' mean future removal in 
ms-speak?  Perhaps after the compile, build.bat should print something like

[Ignore any socket deprecation warnings]

Noting in PR builds that there are new warnings would be great if 
possible. I'd be concerned about it becoming a hard requirement though - 
I much prefer to leave the final decision in the hands of trusted people 
and provide them enough information to make a good decision.


A robot that could scan the appveyor compile log, compare with a 
'current warnings' set, and post a comment requesting a fix and possible 
help sources would be better than leaving them unseen, which I expect is 
the current situation.


--
Terry Jan Reedy

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Steve Dower 

On 29Jun2017 1157, Terry Reedy wrote:

On 6/29/2017 1:54 PM, Steve Dower wrote:
Some warnings are also complicated because of the nature of CPython. 
For example, the socket module exposes deprecated CRT functions (on 
Windows) directly because the API of the socket module promises to 
provide the function directly. Changing to the safer function would 
break the API guarantee (except sometimes it won't... hence 
"complicated").


Thanks for the explanation.  Does 'deprecated' mean future removal in 
ms-speak?  Perhaps after the compile, build.bat should print something like

[Ignore any socket deprecation warnings]


Probably not future removal, considering the vast quantities of legacy 
code that we don't want to break, but renaming is possible (generally 
for standards compliance, such as the timezone->_timezone rename) and 
new platforms (e.g. ARM) are not guaranteed to have the APIs when there 
is essentially no existing code to worry about.


In many (most?) cases, deprecations in MS libraries indicate that there 
are known security issues either inherently in the function itself or 
with many/most uses of it, and there is a better option (e.g. strcpy is 
deprecated in favour of strcpy_s, which requires/infers the length of 
the destination buffer and ensures you never overflow).


I'm still in favour of local suppression over ignoring warnings - we 
just need to promote the macro at 
https://github.com/python/cpython/blob/master/Modules/socketmodule.c#L510 
to somewhere global to make it consistent (and maybe extend it to handle 
other compilers where it matters).


But this is why I'm not against the general idea, but it requires that 
the devguide entry be well written. Let's clean up all the warnings, but 
in the right way, not the way people use when faced with arbitrary rules.


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Victor Stinner 
2017-06-29 19:13 GMT+02:00 Terry Reedy :
> How about compiler warnings (and errors)?

That would be nice to have, but I don't have the bandwidth to handle this goal.

>  When I compile on Windows, there
> are a boatload of orange-yellow warnings.  Some are about using a deprecated
> featured; some about dodgy casts; some (I presume) about other things.
> Should 'no warnings' be a goal?

Last years, I tried to fix a few integer downcast warnings on Windows
64-bit, but they are still a few ones. It's not easy to distinguish
warnings from real bugs (integer overflow).

There are a few open issues about Windows 64-bit warnings.

Segev Finer just wrote an interesting but big PR:
https://github.com/python/cpython/pull/2492

The deprecated warnings is another interesting topic :-)

> I believe you once fixed some, but new commits added more.  Could 'no
> additional warnings' be a CI requirement for merging? I expect that a new
> "How to avoid compiler warnings on Windows" section of the devguide would be
> needed.

My policy is more to start by fixing all warnings, before starting to
check for regressions.

I began by fixing the most common test failures, then references
leaks, and now memory leaks. So it becomes much simpler to check for
regressions on these checks.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot report (almost July)

2017-06-29 Thread Rob Boehne 
Hello,

I¹m new to the list, and contributing to Python specifically, and I¹m
interested in getting master and 3.6 branches building and working
³better² on UNIX.
I¹ve been looking at a problem building 3.6 on HP-UX and see a PR was
merged into master, https://github.com/python/cpython/pull/1351 and I¹d
like to see it applied to 3.6.  I¹m happy to create a PR with a
cherry-picked commit, and/or test.


Rob Boehne 
Senior Software Architect | Datalogics, Inc.
+1.312.853.8351  | [email protected]

datalogics.com  | blogs.datalogics.com
Connect with us: Facebook
 | Twitter
 | LinkedIn
 | YouTube


PROUDLY CELEBRATING 50 YEARS!






On 6/29/17, 10:09 AM, "Python-Dev on behalf of Victor Stinner"
 wrote:

>Hi,
>
>Our buildbots are now even more stable than in my previous buildbot
>report.
>
>Many random failures have been fixed, even if there are still many
>rare random failures (most of them are related to multiprocessing).
>
>Search for issues created by "haypo" (me!) with the Component "Tests"
>to see all remaining issues. I'm trying to open an issue for *each*
>buildbot failure! So yeah, we have a long list of 44 open issues!
>
>Correct me if I'm wrong, but, for the first time, *all reference
>leaks* have been fixed on *all branches* (2.7, 3.5, 3.6 and master),
>on *Linux and Windows*! Before, we mostly focused on the master branch
>(called "default" in Mercurial) on Linux.
>
>I also started to fix a few "memory block" leaks, most (or all?) of
>them should also be fixed (on all branches, on Linux and Windows).
>
>A new policy has been decided (on python-committers): if a commit
>breaks too many buildbots and cannot fixed easily nor quickly, the
>commit will be reverted, just to keep our buildbots "green". It
>doesn't mean that the commit will be rejected forever. It's a
>practical solution to give more time to write a proper fix, take time
>to review it, and not have to be stressed to fix buildbots "ASAP".
>Moreover, keeping green buildbots all the time allows to catch
>regressions more quickly, which ease debug in many ways.
>
>You have be warned! Now I will not hesitate to revert your change if
>you break my little buildbots ;-)
>
>I mostly care of Linux, Windows, macOS and FreeBSD (10 and CURRENT).
>Breaking other platforms is less important, since other platforms
>already have issues, but not enough developers interested to fix them.
>Obviously, I'm interested in keeping more buildbots green, if someone
>shows up and help me to fix remaining issues!
>
>Victor
>___
>Python-Dev mailing list
>[email protected]
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe: 
>https://mail.python.org/mailman/options/python-dev/robb%40datalogics.com

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com