Thank you Wes. I see.
Regards, Tanveer Ahmad ________________________________ From: Wes McKinney <wesmck...@gmail.com> Sent: Thursday, August 22, 2019 5:12:06 PM To: dev@arrow.apache.org Subject: Re: In-memory sorting of plasma objects hi Tanveer, IIUC there is logic for moving data that's managed by Plasma servers between nodes in the Ray project (https://github.com/ray-project/ray) --if you need to move the bytes from one node to another you need to use some kind of messaging / RPC tool. The Ray developers might have some advice -- I think their implementation is specific to Ray's internals which is why we don't have this implemented (yet) natively in Apache Arrow - Wes On Thu, Aug 22, 2019 at 8:34 AM Tanveer Ahmad - EWI <t.ah...@tudelft.nl> wrote: > > Hi, > > > I need some help regarding data exchange between Arrow based plasma shared > memory objects on cluster nodes. > > > I have two Plasma shared memory objects each contains a RecordBatch on > different nodes of a cluster. > > I want to use pandas dataframes or something like that (dask) on a single > node to sort them together. Is there any way to access these Plasma objects > on a single node and sort them in-memory? > > Thanks. > > > Regards, > Tanveer Ahmad
