On Mon, 9 Oct 2023 at 21:57, Nathan <nathan.goldb...@gmail.com> wrote: > > On Mon, Oct 9, 2023 at 2:44 PM Oscar Benjamin <oscar.j.benja...@gmail.com> > wrote: >> Suppose that there is NumPy v1 and that in future there will be NumPy >> v2. Also suppose that there will be two NumPy pickle formats fmtA and >> a future fmtB. One possibility is that NumPy v1 only reads and writes >> fmtA and then NumPy v2 only reads and writes fmtB. One problem with >> this is that when NumPy v2 comes out there is no easy way to convert >> pickles from fmtA to fmtB for compatibility with NumPy v2. Another >> problem with this is that it does not make a nice transition during >> any period of time when both NumPy v1 and v2 might be used in >> different parts of a software stack. > > Doesn't NumpyUnpickler solve this? It will be present in both v1 and v2 and > will allow loading files either np.core or np._core in either version.
I guess that makes it possible in some way to convert formats in either version. I presume though that this still means that a plain pickle.loads() (and any code built on top of such) would fail in v2. >> An alternative is to introduce fmtB as part of the NumPy v1 series. >> NumPy could be changed now so that it can read both fmtA and fmtB but >> by default it would write fmtB which would be designed ahead of time >> so that in future NumPy v2 would be able to read fmtB as well. It >> would also be possible to design it so that fmtB would be readable by >> older versions of NumPy that were released before fmtB was designed. >> >> Then there is a version of NumPy (v1) which can read fmtA and write to >> fmtB. This version of NumPy can be used to convert pickles from fmtA >> to fmtB. Then when NumPy v2 is released it can already read any >> pickles that were generated by the most recent releases of NumPy v1.x. >> Anyone who still has older pickles in fmtA could use NumPy v1 to do >> dumps(loads(f)) which would convert from fmtA to fmtB. >> >> In this scenario the only part that does not work is reading fmtA in >> NumPy v2 which is unavoidable if numpy.core is removed or renamed in >> v2. > > I agree it would have been better to anticipate this and move the > _reconstruct function to np._core many releases ago. Sadly this was not done > and the next release is Numpy 2.0. Well if the next release is NumPy 2.0 then my suggestion does not work. There are alternatives but they might not be worth it at this point. > I also want to emphasize that using pickle like this - to share data between > different python installations - is inherently insecure and should never be > done outside of an organization that fully controls all of the python > installations. In that case, the organization can use NumpyUnpickler. In any > other case, I think it's good to perhaps nudge people away from doing things > like this. Agreed but I guarantee that someone depends on this and is using it in a way that is reasonable for their own purposes. There might not be much to be done about it but someone will experience unexpected breakage and it is worthwhile to contemplate (as you are doing) what can be done to mitigate that. -- Oscar _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
