Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Nathaniel Smith 
On Mar 28, 2017 10:54 AM, "Steve Dower"  wrote:

On 28Mar2017 1035, Paul Moore wrote:

> On 28 March 2017 at 18:05, Glenn Linderman  wrote:
>
>> Somewhere I got the idea that extension authors were supposed to build
>> against the n.m.0 releases, expressly so that the extensions would then be
>> compatible with the whole n.m.x series of releases. Did I dream that?
>>
>
> I've certainly never heard it stated - and I'd think it's a pretty
> annoying requirement to make of extension builders.
> Paul
>

Agreed, we should avoid both additive and subtractive breaking changes to
the binary API within micro versions completely. Additive ones like this
are difficult to catch, unfortunately, and so building against the .0
release is not bad advice. Building with the .0 headers is probably
sufficient.

I wonder if there's a way we can preprocess the headers to define a
baseline? Then we could automatically compare against that and force
explicit decisions leading to public API changes (probably the process of
finishing off the limited API validation could include this task fairly
easily as well).


It wouldn't be as fancy as analyzing the headers, but a much easier and
still useful way to get started would be a test to check no new exported
symbols appear in the shared library during a stable release cycle.

If you want to get quite fancy, libabigail provides a toolkit that can read
debug information and check that all your structs remain the same size etc.
ELF only, but still would catch a lot:

https://sourceware.org/libabigail/manual/libabigail-overview.html

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Steve Dower 

On 28Mar2017 1035, Paul Moore wrote:

On 28 March 2017 at 18:05, Glenn Linderman  wrote:

Somewhere I got the idea that extension authors were supposed to build
against the n.m.0 releases, expressly so that the extensions would then be
compatible with the whole n.m.x series of releases. Did I dream that?


I've certainly never heard it stated - and I'd think it's a pretty
annoying requirement to make of extension builders.
Paul


Agreed, we should avoid both additive and subtractive breaking changes 
to the binary API within micro versions completely. Additive ones like 
this are difficult to catch, unfortunately, and so building against the 
.0 release is not bad advice. Building with the .0 headers is probably 
sufficient.


I wonder if there's a way we can preprocess the headers to define a 
baseline? Then we could automatically compare against that and force 
explicit decisions leading to public API changes (probably the process 
of finishing off the limited API validation could include this task 
fairly easily as well).


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Paul Moore 
On 28 March 2017 at 18:05, Glenn Linderman  wrote:
> Somewhere I got the idea that extension authors were supposed to build
> against the n.m.0 releases, expressly so that the extensions would then be
> compatible with the whole n.m.x series of releases. Did I dream that?

I've certainly never heard it stated - and I'd think it's a pretty
annoying requirement to make of extension builders.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

2017-03-28 Thread Eric Snow 
On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov  wrote:
> Hello,
>
> Let me first clarify, I do not claim this is a bug, I am trying to learn
> python and now I trying to understand yield-from.

Given that you haven't gotten a response here, you may want to take
this over to the core-mentors...@python.org list.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Glenn Linderman 

On 3/28/2017 9:18 AM, Paul Moore wrote:

On 28 March 2017 at 12:24, Miro Hrončok  wrote:

I'd like some clarification on what ABI compatibility we can expect.
  * Should the ABI be stable across patch releases (so calling
PySlice_AdjustIndices from an existing macro would be a bug)?
  * Should the ABI be forward-compatible within a minor release (so modules
built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
  * Or should we expect the ABI to change even across patch releases?

Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.

Somewhere I got the idea that extension authors were supposed to build 
against the n.m.0 releases, expressly so that the extensions would then 
be compatible with the whole n.m.x series of releases. Did I dream that?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Nathaniel Smith 
On Mar 28, 2017 8:29 AM, "Serhiy Storchaka"  wrote:

On 28.03.17 14:24, Miro Hrončok wrote:

> However, recently we found an issue with this approach [1]: an extension
> module built against Python 3.6.1 cannot be run on Python 3.6.0, because
> it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices
> function.
>

The macro expanding to PySlice_AdjustIndices is used only when
Py_LIMITED_API is not defined or is defined to the version equal or greater
the version in which PySlice_AdjustIndices was added.


That's nice, but not sufficient. Py_LIMITED_ABI is cool, but the vast
majority of distributed packages don't use it, and instead rely on the
"unlimited" ABI being forward and backwards compatible within each minor
release. For example, this assumption is hard coded in the wheel format,
which has no way to even describe a wheel that needs 3.6.x with x >= 1.
People uploading packages to pypi use whatever version of 3.6 they have
lying around and assume it will work for everyone downloading.

IMO this is a bug, and depending on how many packages are affected it might
even call for an emergency 3.6.2. The worst case is that we start getting
large numbers of packages uploaded to pypi that claim to be 3.6.0
compatible but that crash like crash with an obscure error when people
download them.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Paul Moore 
On 28 March 2017 at 12:24, Miro Hrončok  wrote:
> I'd like some clarification on what ABI compatibility we can expect.
>  * Should the ABI be stable across patch releases (so calling
> PySlice_AdjustIndices from an existing macro would be a bug)?
>  * Should the ABI be forward-compatible within a minor release (so modules
> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>  * Or should we expect the ABI to change even across patch releases?

Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Serhiy Storchaka 

On 28.03.17 14:24, Miro Hrončok wrote:

However, recently we found an issue with this approach [1]: an extension
module built against Python 3.6.1 cannot be run on Python 3.6.0, because
it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices
function.


The macro expanding to PySlice_AdjustIndices is used only when 
Py_LIMITED_API is not defined or is defined to the version equal or 
greater the version in which PySlice_AdjustIndices was added.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What version is an extension module binary compatible with

2017-03-28 Thread Miro Hrončok 

Hi,

as per [0], ABI of the C API is generally not stable and the binary 
compatibility may break between versions. It is hard from the text to 
know whether it talks about minor versions (such as 3.6 vs 3.5) or patch 
versions (such as 3.6.1 vs 3.6.0).


In Fedora we currently only keep track about the minor version 
dependency. I.e. an RPM package with a Python module depends on Python 
3.6, not specifically on Python 3.6.1.


However, recently we found an issue with this approach [1]: an extension 
module built against Python 3.6.1 cannot be run on Python 3.6.0, because 
it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices function.


I'd like some clarification on what ABI compatibility we can expect.
 * Should the ABI be stable across patch releases (so calling 
PySlice_AdjustIndices from an existing macro would be a bug)?
 * Should the ABI be forward-compatible within a minor release (so 
modules built for 3.6.0 should be usable with 3.6.1, but not vice versa)?

 * Or should we expect the ABI to change even across patch releases?

It would be nice to say this explicitly in the docs ([0] or another 
suitable place).



[0] https://docs.python.org/3/c-api/stable.html
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1435135

Thanks for clarification,
On behalf of the Fedora Python SIG,
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Misc/NEWS entries for Python 3.7a1

2017-03-28 Thread Terry Reedy 

On 3/28/2017 8:49 AM, INADA Naoki wrote:


Currently, changelog of Python 3.7a1 [1] contains changes between
3.6b1 and 3.7a1.


I think the changelog for x.(y+1).0a1 should start with the released 
x.y.0.  This used to be the case (with perhaps a few exceptions) when 
x.y.0 was not branched off until the release candidate (or maybe not 
until the releas), and people were asked to stop pushing enhancements 
between beta1 and the release.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Misc/NEWS entries for Python 3.7a1

2017-03-28 Thread Ned Deily 
On Mar 28, 2017, at 08:49, INADA Naoki  wrote:
> Currently, changelog of Python 3.7a1 [1] contains changes between
> 3.6b1 and 3.7a1.
> So lot's of bugfixes are listed twice or more in changelog.
> For example, "bpo-28258: Fixed build with Estonian locale..." are
> listed under 3.5.3rc1,
> 3.6.0b2 and 3.7.0a1.
> 
> [1] https://docs.python.org/3.7/whatsnew/changelog.html#changelog
[...]

Thanks for noticing. Misc/NEWS is always somewhat problematic.  As you probably 
know, the Core Workflow SIG, led by Brett, is working on a long-term solution 
to generating Misc/NEWS, a solution that should be available soon.  One of the 
duties of the release manager is to "edit" Misc/NEWS; I was planning to wait 
for the new Misc/NEWS solution and for more of the conversion to Git/GitHub to 
settle to do anything major to the master (i.e. 3.7) version.  There have 
already been some major merge mistakes for the 3.6.x Misc/NEWS.  I would 
recommend not to worry too much about master's Misc/NEWS right now.  I may do 
some cleaning up before the new Misc/NEWS process is introduced but I will also 
be reviewing it prior to each of the preview releases, which start later this 
year.

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

2017-03-28 Thread Terry Reedy 

Steven, thanks for verifying bug on *nix.

On 3/28/2017 2:00 AM, Martin Panter wrote:

On 28 March 2017 at 03:11, Steven D'Aprano  wrote:

On Mon, Mar 27, 2017 at 10:33:44PM -0400, Terry Reedy wrote:

https://bugs.python.org/issue29926 was opened as an IDLE issue, which
means that most watching the new issues list would ignore it.  But I
think it is an issue with _thread.interrupt_main (which IDLE calls in
respond to ^C) not interrupting time.sleep(n) in main thread*.  I tested
on Windows, don't know yet about OP.  Since there is no Expert's Index
listing for _thread (or threading), I am asking here for someone who
knows anything to take a look.

*

time.sleep(10)



<... remainder of 10 seconds pass>
KeyboardInterrupt



I get similar behaviour under Linux. I don't have the debug print, but
the KeyboardInterrupt doesn't interrupt the sleep until the 10 seconds
are up.


Looking at the implementation, _thread.interrupt_main just calls
PyErr_SetInterrupt. It doesn’t appear to send a signal. I played with
“strace” and couldn’t see any evidence of a signal. I guess it just
sets a flag that will be polled. To actually interrupt the “sleep”
call, you might need to use “pthread_kill” or similar (at least on
Unix).


I copied this to the issue.  Eryk Sun suggested a patch for Windows, 
(and the possibility of using pthread_kill).  Can you possibly do one 
for *nix?  This is out of my ballpark, but the bug (relative to console 
behavior) is a nuisance.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Misc/NEWS entries for Python 3.7a1

2017-03-28 Thread INADA Naoki 
Hi.

Currently, changelog of Python 3.7a1 [1] contains changes between
3.6b1 and 3.7a1.
So lot's of bugfixes are listed twice or more in changelog.
For example, "bpo-28258: Fixed build with Estonian locale..." are
listed under 3.5.3rc1,
3.6.0b2 and 3.7.0a1.

[1] https://docs.python.org/3.7/whatsnew/changelog.html#changelog

This has two problems:

  * The changelog is longer than necessary.
  * If bpo- is fixed in 3.7a1, people may think the bug exists in
3.6.0, even if the
bug is fixed in 3.6b2 too.

Since we stopped merging 3.6 -> master, I suggest to remove such duplicates.
There are two ways:


# A: 3.7a1 -> 3.6.0 -> 3.6.0rc2 ... -> 3.6a1 -> 3.5.0 ...

This may be what people expect.
In this case, we will remove:

* 3.6.1(rc*) from changelog
* duplicated entries in 3.7.0 (which fixed before 3.6.0)


# B: 3.7a1 -> 3.6b1 -> ... 3.6a1 -> 3.5b1 ...

This reflects our branch model.
In this case, we will remove:

* All 3.6 versions after 3.6b1


How do you think?

Regards,
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Issue with _thread.interrupt_main (29926)

2017-03-28 Thread Martin Panter 
On 28 March 2017 at 03:11, Steven D'Aprano  wrote:
> On Mon, Mar 27, 2017 at 10:33:44PM -0400, Terry Reedy wrote:
>> https://bugs.python.org/issue29926 was opened as an IDLE issue, which
>> means that most watching the new issues list would ignore it.  But I
>> think it is an issue with _thread.interrupt_main (which IDLE calls in
>> respond to ^C) not interrupting time.sleep(n) in main thread*.  I tested
>> on Windows, don't know yet about OP.  Since there is no Expert's Index
>> listing for _thread (or threading), I am asking here for someone who
>> knows anything to take a look.
>>
>> *
>> >>> time.sleep(10)
>> 
>> 
>> <... remainder of 10 seconds pass>
>> KeyboardInterrupt
>
>
> I get similar behaviour under Linux. I don't have the debug print, but
> the KeyboardInterrupt doesn't interrupt the sleep until the 10 seconds
> are up.

Looking at the implementation, _thread.interrupt_main just calls
PyErr_SetInterrupt. It doesn’t appear to send a signal. I played with
“strace” and couldn’t see any evidence of a signal. I guess it just
sets a flag that will be polled. To actually interrupt the “sleep”
call, you might need to use “pthread_kill” or similar (at least on
Unix).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com