On Mon, Oct 9, 2023 at 3:12 PM Oscar Benjamin <oscar.j.benja...@gmail.com> wrote:
> On Mon, 9 Oct 2023 at 21:57, Nathan <nathan.goldb...@gmail.com> wrote: > > > > On Mon, Oct 9, 2023 at 2:44 PM Oscar Benjamin < > oscar.j.benja...@gmail.com> wrote: > >> Suppose that there is NumPy v1 and that in future there will be NumPy > >> v2. Also suppose that there will be two NumPy pickle formats fmtA and > >> a future fmtB. One possibility is that NumPy v1 only reads and writes > >> fmtA and then NumPy v2 only reads and writes fmtB. One problem with > >> this is that when NumPy v2 comes out there is no easy way to convert > >> pickles from fmtA to fmtB for compatibility with NumPy v2. Another > >> problem with this is that it does not make a nice transition during > >> any period of time when both NumPy v1 and v2 might be used in > >> different parts of a software stack. > > > > Doesn't NumpyUnpickler solve this? It will be present in both v1 and v2 > and will allow loading files either np.core or np._core in either version. > > I guess that makes it possible in some way to convert formats in > either version. I presume though that this still means that a plain > pickle.loads() (and any code built on top of such) would fail in v2. > In Numpy2.0 you would see a deprecation warning about the path in the pickle file but no crash. Eventually, when we finally remove the stub np.core, you would see a crash. However, one thing we can do now is, for that one particular symbol that we know is going to be in every pickle file and probably never elsewhere, is intercept that one import and instead of generating a generic warning about np.core being deprecated, we instead make that specific version of the deprecation warning mentions NumpyUnpickler. I'll make sure this gets done. We *could* just allow that import to happen without a warning, but then we're stuck keeping np.core around even longer and we also will still generate a deprecation warning for an import from np.core if the pickle file happens to include any other numpy types that might generate imports in np.core. > > >> An alternative is to introduce fmtB as part of the NumPy v1 series. > >> NumPy could be changed now so that it can read both fmtA and fmtB but > >> by default it would write fmtB which would be designed ahead of time > >> so that in future NumPy v2 would be able to read fmtB as well. It > >> would also be possible to design it so that fmtB would be readable by > >> older versions of NumPy that were released before fmtB was designed. > >> > >> Then there is a version of NumPy (v1) which can read fmtA and write to > >> fmtB. This version of NumPy can be used to convert pickles from fmtA > >> to fmtB. Then when NumPy v2 is released it can already read any > >> pickles that were generated by the most recent releases of NumPy v1.x. > >> Anyone who still has older pickles in fmtA could use NumPy v1 to do > >> dumps(loads(f)) which would convert from fmtA to fmtB. > >> > >> In this scenario the only part that does not work is reading fmtA in > >> NumPy v2 which is unavoidable if numpy.core is removed or renamed in > >> v2. > > > > I agree it would have been better to anticipate this and move the > _reconstruct function to np._core many releases ago. Sadly this was not > done and the next release is Numpy 2.0. > > Well if the next release is NumPy 2.0 then my suggestion does not > work. There are alternatives but they might not be worth it at this > point. > > > I also want to emphasize that using pickle like this - to share data > between different python installations - is inherently insecure and should > never be done outside of an organization that fully controls all of the > python installations. In that case, the organization can use > NumpyUnpickler. In any other case, I think it's good to perhaps nudge > people away from doing things like this. > > Agreed but I guarantee that someone depends on this and is using it in > a way that is reasonable for their own purposes. There might not be > much to be done about it but someone will experience unexpected > breakage and it is worthwhile to contemplate (as you are doing) what > can be done to mitigate that. > > -- > Oscar > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: nathan12...@gmail.com >
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
