Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On Fri, Aug 24, 2018 at 4:00 PM, Stephan Hoyer wrote:
> On Fri, Aug 24, 2018 at 3:14 PM Nathaniel Smith wrote:
>>
>> Yeah, the reason warnings are normally recommended is because
>> normally, you want to make it easy to silence. But this is the rare
>> case where I didn't want to make it easy to silence, so I didn't
>> suggest using a warning :-).
>>
>> Calling warnings.warn (or the C equivalent) is also very expensive,
>> even if the warning ultimately isn't displayed. I guess we could do
>> our own tracking of whether we've displayed the warning yet, and only
>> even attempt to issue it once, but that partially defeats the purpose
>> of using warnings in the first place.
>
>
> I thought the suggestion was to issue a warning when
> np.enable_experimental_array_function() is called. I agree that it's a
> non-starter to issue it every time an __array_function__ method is called --
> warnings are way too slow for that.
If our protection against uninformed usage is a Big Obnoxious
Warning(tm), then I was imagining that we could simplify by dropping
enable_experimental_array_function entirely. Doesn't make a big
difference either way though.
> People can redirect stderr, so we're really not stopping anyone from
> silencing things by doing it in a non-standard way. We're just making it
> annoying and non-standard. Developers could even run Python in a subprocess
> and filter out all the warnings -- there's really nothing we can do to stop
> determined abusers of this feature.
>
> I get that you want to make this annoying and non-standard, but this is too
> extreme for me. Do you seriously imagine that we'll consider ourselves
> beholden in the future to users who didn't take us at our word?
Let's break that question down into two parts:
1. if we do find ourselves in a situation where changing this would
break lots of users, will we consider ourselves beholden to them?
2. is it plausible that we'll find ourselves in that situation?
For the first question, I think the answer is... yes? We constantly
bend over backwards to try to avoid breaking users. Our deprecation
policy explicitly says that it doesn't matter what we say in the docs,
the only thing that matters is whether a change breaks users. And to
make things more complicated, it's easy to imagine scenarios where the
people being broken aren't the ones who had a chance to read the docs
– e.g. if a major package starts relying on __array_function__, then
it's all *their* users who we'd be breaking, even though they had
nothing to do with it. If any of
{tensorflow/astropy/dask/sparse/sklearn/...} did start relying on
__array_function__ for normal functionality, then *of course* that
would come up in future discussions about changing __array_function__,
and *of course* it would make us reluctant to do that. As it should,
because breaking users is bad, we should try to avoid ending up in
situations where that's what we have to do, even if we have a NEP to
point to to justify it.
But... maybe it's fine anyway, because this situation will never come
up? Obviously I hope that our downstreams are all conscientious, and
friendly, and take good care of their users, and would never create a
situation like that. I'm sure XArray won't :-). But... people are
busy, and distracted, and have pressures to get something shipped, and
corners get cut. Companies *care* about what's right, but they mostly
only *do* the minimum they have to. (Ask anyone who's tried to get
funding for OSS...) Academics *care* about what's right, but they just
don't have time to care much. So yeah... if there's a quick way to
shut up the warning and make things work (or seem to work,
temporarily), versus doing things right by talking to us, then I do
think people might take the quick hack.
The official Tensorflow wheels flat out lie about being manylinux
compatible, and the Tensorflow team has never talked to anyone about
how to fix this, they just upload them to PyPI and leave others get to
deal with the fallout [1]. That may well be what's best for their
users, I don't know. But stuff like this is normal, it happens all the
time, and if someone does it with __array_function__ then we have no
leverage.
-n
[1] https://github.com/tensorflow/tensorflow/issues/8802#issuecomment-401703703
--
Nathaniel J. Smith -- https://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On 29/08/18 10:37, Nathaniel Smith wrote: it's easy to imagine scenarios where the people being broken aren't the ones who had a chance to read the docs – e.g. if a major package starts relying on __array_function__, then it's all*their* users who we'd be breaking, even though they had nothing to do with it. This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages. Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP? Matti ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
> On 29. Aug 2018, at 11:44, Matti Picus wrote: > > On 29/08/18 10:37, Nathaniel Smith wrote: >> it's easy to imagine scenarios where the >> people being broken aren't the ones who had a chance to read the docs >> – e.g. if a major package starts relying on __array_function__, then >> it's all*their* users who we'd be breaking, even though they had >> nothing to do with it. > This is a packaging problem. This proposal is intended for use by other > "major packages", not so much for end-users. We would have much more trouble > if we were proposing a broad change to something like indexing or the random > number module (see those NEPs). If we break one of those major packages, it > is on them to pin the version of NumPy they can work with. In my opinion very > few end users will be implementing their own ndarray classes with > `__array_function__`. While we will get issue reports, we can handle them > much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users > to complain to those packages. One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion. Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime. > > Other than adding a warning, I am not sure what the concrete proposal is > here. To not accept the NEP? > Matti > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them? I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis. Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts). Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out. This is different from how Numpy has operated in the past, but that might be ok. >> 2. is it plausible that we'll find ourselves in that situation? > Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime Yup. Same for Dask. We're pretty accustomed to this. On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi wrote: > > On 29. Aug 2018, at 11:44, Matti Picus wrote: > > > > On 29/08/18 10:37, Nathaniel Smith wrote: > >> it's easy to imagine scenarios where the > >> people being broken aren't the ones who had a chance to read the docs > >> – e.g. if a major package starts relying on __array_function__, then > >> it's all*their* users who we'd be breaking, even though they had > >> nothing to do with it. > > This is a packaging problem. This proposal is intended for use by other > "major packages", not so much for end-users. We would have much more > trouble if we were proposing a broad change to something like indexing or > the random number module (see those NEPs). If we break one of those major > packages, it is on them to pin the version of NumPy they can work with. In > my opinion very few end users will be implementing their own ndarray > classes with `__array_function__`. While we will get issue reports, we can > handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem > and urge users to complain to those packages. > > One thing that might help here is nightly or continuous CI builds of the > NumPy wheels. This would be good, as we could test it in CI, and fix it > when it comes up. But I guess that’s another discussion. > > Personally, for as long as this protocol is experimental, I’ll add a > warning in the docs of sparse as well; saying this might disappear anytime. > > > > > Other than adding a warning, I am not sure what the concrete proposal is > here. To not accept the NEP? > > Matti > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@python.org > > https://mail.python.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
HI All, On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13. One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.) All the best, Marten On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin wrote: > >> 1. if we do find ourselves in a situation where changing this would > break lots of users, will we consider ourselves beholden to them? > > I think that it would be useful for Numpy's continued evolution to develop > the ability to include code on a provisional basis. Other projects do this > and they just have big bold "Experimental" notes everywhere that a user > might go to learn about the functionality (docs, docstrings, blogposts). > Some users will definitely get burned, yes, but the alternative is to burn > all other users a little bit by moving too slowly and not trying things out. > > This is different from how Numpy has operated in the past, but that might > be ok. > > >> 2. is it plausible that we'll find ourselves in that situation? > > > Personally, for as long as this protocol is experimental, I’ll add a > warning in the docs of sparse as well; saying this might disappear anytime > > Yup. Same for Dask. We're pretty accustomed to this. > > On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi > wrote: > >> > On 29. Aug 2018, at 11:44, Matti Picus wrote: >> > >> > On 29/08/18 10:37, Nathaniel Smith wrote: >> >> it's easy to imagine scenarios where the >> >> people being broken aren't the ones who had a chance to read the docs >> >> – e.g. if a major package starts relying on __array_function__, then >> >> it's all*their* users who we'd be breaking, even though they had >> >> nothing to do with it. >> > This is a packaging problem. This proposal is intended for use by other >> "major packages", not so much for end-users. We would have much more >> trouble if we were proposing a broad change to something like indexing or >> the random number module (see those NEPs). If we break one of those major >> packages, it is on them to pin the version of NumPy they can work with. In >> my opinion very few end users will be implementing their own ndarray >> classes with `__array_function__`. While we will get issue reports, we can >> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem >> and urge users to complain to those packages. >> >> One thing that might help here is nightly or continuous CI builds of the >> NumPy wheels. This would be good, as we could test it in CI, and fix it >> when it comes up. But I guess that’s another discussion. >> >> Personally, for as long as this protocol is experimental, I’ll add a >> warning in the docs of sparse as well; saying this might disappear anytime. >> >> > >> > Other than adding a warning, I am not sure what the concrete proposal >> is here. To not accept the NEP? >> > Matti >> > ___ >> > NumPy-Discussion mailing list >> > NumPy-Discussion@python.org >> > https://mail.python.org/mailman/listinfo/numpy-discussion >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever. On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk < m.h.vankerkw...@gmail.com> wrote: > HI All, > > On the backwards compatibility: from an astropy perspective, I would > expect that the introduction of `__array_function__` implies a guarantee > that the *functionality* it provides will remain, i.e., that it will > continue to be possible to override, say, concatenate. It is not a big deal > if the actual implementation changes (say, `__array_concatenate__` is > introduced) and we need to do some version-dependent magic to ensure that > our users do not notice the change; we did the same with `__array_ufunc__` > - the wrap/prepare machinery will be gone only in the next version of > astropy, when our minimum version for numpy reaches 1.13. > > One thing perhaps worth remembering that what really matters is not so > much `__array_function__` but the set of functions that actually implement > the override. It would seem unlikely this would be the complete numpy API > on the first go; perhaps it is an idea to be somewhat specific about which > part we start with? My vote would go towards array manipulation routines > like concatenate. I would also suggest to avoid those functions for which > ufunc implementations would seem quite possible (i.e., avoid things like > median, etc.) > > All the best, > > Marten > > On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin > wrote: > >> >> 1. if we do find ourselves in a situation where changing this would >> break lots of users, will we consider ourselves beholden to them? >> >> I think that it would be useful for Numpy's continued evolution to >> develop the ability to include code on a provisional basis. Other projects >> do this and they just have big bold "Experimental" notes everywhere that a >> user might go to learn about the functionality (docs, docstrings, >> blogposts). Some users will definitely get burned, yes, but the >> alternative is to burn all other users a little bit by moving too slowly >> and not trying things out. >> >> This is different from how Numpy has operated in the past, but that might >> be ok. >> >> >> 2. is it plausible that we'll find ourselves in that situation? >> >> > Personally, for as long as this protocol is experimental, I’ll add a >> warning in the docs of sparse as well; saying this might disappear anytime >> >> Yup. Same for Dask. We're pretty accustomed to this. >> >> On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi >> wrote: >> >>> > On 29. Aug 2018, at 11:44, Matti Picus wrote: >>> > >>> > On 29/08/18 10:37, Nathaniel Smith wrote: >>> >> it's easy to imagine scenarios where the >>> >> people being broken aren't the ones who had a chance to read the docs >>> >> – e.g. if a major package starts relying on __array_function__, then >>> >> it's all*their* users who we'd be breaking, even though they had >>> >> nothing to do with it. >>> > This is a packaging problem. This proposal is intended for use by >>> other "major packages", not so much for end-users. We would have much more >>> trouble if we were proposing a broad change to something like indexing or >>> the random number module (see those NEPs). If we break one of those major >>> packages, it is on them to pin the version of NumPy they can work with. In >>> my opinion very few end users will be implementing their own ndarray >>> classes with `__array_function__`. While we will get issue reports, we can >>> handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem >>> and urge users to complain to those packages. >>> >>> One thing that might help here is nightly or continuous CI builds of the >>> NumPy wheels. This would be good, as we could test it in CI, and fix it >>> when it comes up. But I guess that’s another discussion. >>> >>> Personally, for as long as this protocol is experimental, I’ll add a >>> warning in the docs of sparse as well; saying this might disappear anytime. >>> >>> > >>> > Other than adding a warning, I am not sure what the concrete proposal >>> is here. To not accept the NEP? >>> > Matti >>> > ___ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussi
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin wrote: > > On the backwards compatibility: from an astropy perspective, I would > expect that the introduction of `__array_function__` implies a guarantee > that the *functionality* it provides will remain, > > My guess is that you wouldn't have this expectation if Numpy released this > feature with explicit "Experimental" warnings attached to it. In that > case astropy might just wait before adopting it until that label was > dropped. Projects that were more risk-friendly would then take on the > burden of trying it out, working out some kinks, and then hopefully in a > version or two the "Experimental" label would be dropped and astropy would > step in and adopt it more safely. > > Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
> Well, I guess I'll be proving Nathaniel right: I would *definitely* start > using __array_function__ in astropy - not being able to concatenate Quantity > and other instances which use it has been a long-standing pain. That's fair enough, but I think Matt's point still stands. Any given project can weigh the benefits of early adoption against the risks, considering its need for a new feature and its resources available to deal with possible future changes. An "Experimental" designation gives numpy a little space to innovate and lets individual projects opt in. Without that space, numpy has to take on all the burden of continuity instead of sharing that burden with the community of projects that opt in to early adoption. Best, Dan Daniel B. Allan, Ph.D Associate Computational Scientist, Brookhaven National Lab (631) 344-3281 (no voicemail set up) From: NumPy-Discussion on behalf of Marten van Kerkwijk Sent: Wednesday, August 29, 2018 10:43:55 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin mailto:mrock...@gmail.com>> wrote: > On the backwards compatibility: from an astropy perspective, I would expect > that the introduction of `__array_function__` implies a guarantee that the > *functionality* it provides will remain, My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings attached to it. In that case astropy might just wait before adopting it until that label was dropped. Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely. Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that - the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.) -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented). -- Marten p.s. And, yes, do count me in for helping to back up that commitment! ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On Wed, Aug 29, 2018, 02:44 Matti Picus wrote: > On 29/08/18 10:37, Nathaniel Smith wrote: > > it's easy to imagine scenarios where the > > people being broken aren't the ones who had a chance to read the docs > > – e.g. if a major package starts relying on __array_function__, then > > it's all*their* users who we'd be breaking, even though they had > > nothing to do with it. > This is a packaging problem. This proposal is intended for use by other > "major packages", not so much for end-users. We would have much more > trouble if we were proposing a broad change to something like indexing > or the random number module (see those NEPs). If we break one of those > major packages, it is on them to pin the version of NumPy they can work > with. In my opinion very few end users will be implementing their own > ndarray classes with `__array_function__`. While we will get issue > reports, we can handle them much as we do the MKL or OpenBLAS ones - > pinpoint the problem and urge users to complain to those packages. > > Other than adding a warning, I am not sure what the concrete proposal is > here. To not accept the NEP? > The proposal is just that while the NEP is considered experimental and provisional, we should use some kind of technical measures to discourage use in a non-experimental settings. We want to stay in control of when it escapes the lab, and docs alone, or trivially disableable messages, aren't a very effective way to do that. -n > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk < m.h.vankerkw...@gmail.com> wrote: > Absolutely fine to have to deal with future chances - my main point is > that by accepting the NEP, I think numpy is committing to provide some way > to override whatever functions __array_function__ is introduced for, i.e., > we cannot reasonably go back to not providing any way to override such a > function (but by marking the feature experimental, we *can* change the way > the override of any given function is implemented). > I'm not entirely sure that we're ready to make this commitment yet, but I think this ties into our decision about what sort of user opt-ins to require. I'd like to structure this decision in terms of the types of breaking changes we contemplate possibly making in this future with regards to this protocol: 1. Abandoning this approach of allowing overrides for NumPy's high level API entirely, in favor of requiring the use of a dedicated namespace like numpy.api. This might arise if we decide that the overhead of checking for __array_function__ attributes is too much, or if it's simply too confusing to allow NumPy functions to be overloaded in arbitrary ways. 2. Changing the general __array_function__ protocol in a breaking way, e.g., to eliminate the "types" argument. 3. Replacing __array_function__ with another override mechanism, either (a) in general (e.g., __array_function_v2__ without the "types" argument) or (b) for specific functions (e.g., switching to a more specific protocol like __array_ufunc__). 4. Removing the possibility of overriding specific functions altogether, because we want to require using a more generic interface (e.g., to require overriding np.stack implicitly via np.expand_dims/np.concatenate instead). Possible changes (1) or (2) would be good reasons for requiring end-users to make an intentional choice to enable __array_function__. Either of these changes would break everyone who uses the protocol. Concern about change (1) would be a particularly good reason to require an opt-in, because it's concerned about impact on users who don't use any new functionality. Possible changes (3) or (4) would be annoying to users, but in my mind would not justify requiring an explicit opt-in. Both could be achieved without immediately breaking users by making use of a standard deprecation cycle, i.e., by still calling __array_function__ but issuing a FutureWarning. The main reason for requiring the explicit opt-in would be if we don't want to need to bother with a deprecation cycle. To be clear, with the exception of change (3)(b), I don't think any of these are particularly likely or desirable outcomes. So in my personal opinion, I don't think we need the explicit opt-in. But I'm willing to make to go ahead with requiring the opt-in if that's what it takes to get everyone on board -- as long as we don't make the opt-in too onerous in a way that imposes hard limits on this protocol can be used. (I would be OK with an explicit Python API for enabling this feature that raises a FutureWarning when it's called, without any API for turning it off.) Best, Stephan ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith wrote: > The official Tensorflow wheels flat out lie about being manylinux > compatible, and the Tensorflow team has never talked to anyone about > how to fix this, they just upload them to PyPI and leave others get to > deal with the fallout [1]. That may well be what's best for their > users, I don't know. But stuff like this is normal, it happens all the > time, and if someone does it with __array_function__ then we have no > leverage. Yes, this is a good example of how library authors sometimes break rules imposed by their dependencies when not enforced by technical means. I'd like to think the TensorFlow team was making an intentional choice here and is willing to live with the consequence of their decision. Unfortunately right now many of those consequences are imposed on others, so this was certainly an inconsiderate choice. I don't know if they were aware of these costs that they are imposing on the rest of the Python ecosystem. I don't see any mention in PEP-513 about the consequences of lying about manylinux compatibility -- even though arguably such warnings should not be necessary. One lesson we might take from this is documentation is essential: we should have more prominent warnings in NEP-18 (given that it's also likely to serve as a form of user-facing docs) about how experimental this protocol is, and about the specific types of changes we anticipate in the future (e.g., see my other email). I also hope that Python packaging maintainers would not hesitate to break this abuse of the manylinux1 tag, e.g., if they wanted to add compatibility checks on wheels when uploaded to PyPI. ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
