On Fri, Aug 14, 2020 at 11:56 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > > > > > Regarding Plasma, you're right we should have started this conversation > > earlier! The way it's being developed in Ray currently isn't useful as a > > standalone project. We realized that tighter integration with Ray's object > > lifetime tracking could be important, and removing IPCs and making it a > > separate thread in the same process as our scheduler could make a big > > difference for performance. Some of these optimizations wouldn't be easy > > without a tight integration, so there are some trade-offs here. > > So I guess the question is whether it is worth continuing to try to > maintain a sepearate version of plasma within the Arrow repo? >
What isn't clear is whether the Plasma that's in Ray is usable in a C++ library context (e.g. what we currently ship as libplasma-dev e.g. on Ubuntu/Debian). That seems still useful, but if the project isn't being actively maintained / developed (which, given the series of stale PRs over the last year or two, it doesn't seem to be) it's unclear whether we want to keep shipping it. > On Tue, Jul 21, 2020 at 9:28 AM Robert Nishihara <robertnishih...@gmail.com> > wrote: > > > Hi all, > > > > Regarding Plasma, you're right we should have started this conversation > > earlier! The way it's being developed in Ray currently isn't useful as a > > standalone project. We realized that tighter integration with Ray's object > > lifetime tracking could be important, and removing IPCs and making it a > > separate thread in the same process as our scheduler could make a big > > difference for performance. Some of these optimizations wouldn't be easy > > without a tight integration, so there are some trade-offs here. > > > > Regarding the Python serialization format, I agree with Antoine that it > > should be deprecated. We began developing it before pickle 5, but now that > > pickle 5 has taken off, it makes less sense (it's useful in its own right, > > but at the end of the day, we were interested in it as a way to serialize > > arbitrary Python objects). > > > > -Robert > > > > On Sun, Jul 12, 2020 at 5:26 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > I'll add deprecation warnings to the pyarrow.serialize functions in > > > question, it will be pretty simple. > > > > > > On Sun, Jul 12, 2020, 6:34 PM Neal Richardson < > > neal.p.richard...@gmail.com > > > > > > > wrote: > > > > > > > This seems like something to investigate after the 1.0 release. > > > > > > > > Neal > > > > > > > > On Sun, Jul 12, 2020 at 11:53 AM Antoine Pitrou <anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > I'd certainly like to deprecate our custom Python serialization > > format, > > > > > and using pickle protocol 5 instead is a very good idea. > > > > > > > > > > We can probably keep it in 1.0 while raising a FutureWarning. > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > Le 12/07/2020 à 19:22, Wes McKinney a écrit : > > > > > > It appears that the Ray developers have decided to fork Plasma and > > > > > > decouple from the Arrow codebase: > > > > > > > > > > > > https://github.com/ray-project/ray/pull/9154 > > > > > > > > > > > > This is a disappointing development to occur without any discussion > > > on > > > > > > this mailing list but given the lack of development activity on > > > Plasma > > > > > > I would like to see how others in the community would like to > > > proceed. > > > > > > > > > > > > It appears additionally that the Union-based serialization format > > > > > > implemented by arrow/python/serialize.h and the > > pyarrow/serialize.py > > > > > > has been dropped in favor of pickle5. If there is not value in > > > > > > maintaining this code then it would probably be preferable for us > > to > > > > > > remove this from the codebase. > > > > > > > > > > > > Thanks, > > > > > > Wes > > > > > > > > > > > > > > > > > > > >
