On Fri, 2020-04-10 at 18:19 +0200, Ralf Gommers wrote:
> On Fri, Apr 10, 2020 at 3:03 PM Sebastian Berg <
> sebast...@sipsolutions.net>
> wrote:
>
> > On Fri, 2020-04-10 at 12:27 +0200, Ralf Gommers wrote:
> > > > 3. I added features to allow transitioning::
> > > >
> > > > get_array_module(*arrays, modules="numpy",
> > > > future_modules=("dask.array", "cupy"),
> > > > fallback="warn")
> > > >
> > > > Will give FutureWarning/DeprecationWarning where necessary,
> > > > in
> > > the
> > > > above "numpy" is supported, dask and cupy are supported but
> > > > not
> > > > enabled by default. `None` works to say "all modules".
> > > > Once the transition is done, just move dask and cupy into
> > > `modules`
> > > > and remove `fallback=None`.
> > > >
> > >
> > > So future_modules explicitly excludes compatible libraries that
> > > are
> > > not
> > > listed. Why would you want anyone to do that? I don't understand
> > > "supported
> > > but not enabled", and it looks undesirable to me to special-case
> > > any
> > > library in this mechanism.
> >
> > We hav two (or three) types of modules (either could be "all").
> >
>
> I think we only have modules that implement __array_module__, and
> ones that
> don't.
>
>
> > 1. Supported modules that we dispatch to.
> > 2. Modules that are supported but will be dispatched to by default
> > only
> > in the future. So if the user got a future_module, they will get
> > a
> > FutureWarning. They have to chose to cast the inputs or opt-in
> > to
> > the future behaviour.
> > 3. Unsupported modules: If this is resolved it is an error. I
> > currently
> > assume that this does not need to be a negative list.
> >
> > You need to distinguish those somehow, since you need a way to
> > transition. Even if you expect that modules would always be *all*
> > modules, `numpy` is still the only accepted module originally.
> >
> > So, as I said, `future_modules` is only about transitioning and
> > enabling `FutureWarning`s. Does not have to live there, but we need
> > a
> > way to transition.
> >
>
> Sorry, I still don't get it - transition what? You seem to be
> operating on
> the assumption that the users of get_array_module want or need to
> control
> which numpy-like libraries they allow and which they don't. That
> seems
> fundamentally wrong. How would you treat, for example, an array
> library
> that is developed privately inside some company?
>
Well, you still need to transition from NumPy -> allow everything, so
for now please just ignore that part if you like and use/assume:
get_array_module(...,
modules="numpy", future_modules=None, fallback="warn")
during the transition, and:
get_array_module(...)
after it. After all this is a draft-project right now, so it is just as
much about trying out what can be done.
It is not unlikely that this transition burden will be put more on the
library in any case, but it shows that it can be done.
As to my "fundamentally wrong" assumption. Should libraries goal be to
support everything? Definitely!
But... I do not want to make that decision for libraries, so I if
library authors tell me that they have no interest in it, all the
better. Until then I am more than happy to keep that option on the
table. If just as a thought for library authors to consider their
options.
Possible, brainstorming, reasons could be:
1. Say I currently heavily use cython code, so I am limited to NumPy
(or at least arrays that can expose a buffer/`__array_interface__`).
Now if someone adds a CUDA implementation, I would support cupy arrays,
but not distributed arrays.
I admit maybe checking that at function entry like this is the wrong
approach there.
2. To limit to certain types is to say "We know (and test) that our
library works with xarray, Dask, NumPy, and CuPy". Now you can say that
is also a misconception, because if you stick to just NumPy API you
should know that it will "just work" with everything. But in practice
it seems like it might happen?
In that case you may want to actually allow any odd array and just put
a warning, a bit like the transition warnings I put in for testing.
---
There are two other things I am wondering about.
1. Subclasses may want to return their superclasses module (even by
default?), in which case their behaviour depends on the superclass
module behaviour. Further a library would need to use `np.asanyarray()`
to prevent the subclass from being cast to the superclass.
2. There is one transition that does not quite exists. What if an
array-like starts implementing or expands `array-module`?
That seems fine, but in that case the array-like will have to provide
the `opt-in` context manager with a FutureWarning.
The transition from no `__array_module__` to implementing it may need
some thought, but I expect it is fine: The array-like simply always
gives a FutureWarning, although it cannot know what will actually
happen in the future (no change, error, or array-like takes control).
- Sebastian
> Cheers,
> Ralf
>
>
>
> > These options do not have to be handled by us, it only helps here
> > with
> > having context managers to opt-in to new behaviour, and maybe to
> > get an
> > idea for how transitions can look like.
> > Alternatively, we could all to create project specific context
> > managers
> > to do the same and avoid possible scoping issues even more.
> >
> > - Sebastian
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
signature.asc
Description: This is a digitally signed message part
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
