Hi Mike, 1. I think yes, though we'd need to turn off the automatic LRU eviction that happens when the store fills up.
3. I think there are some edge cases and it depends what is in your DataFrame, but at least if it consists of numerical data then the two representations should use the same underlying data in shared memory. On Thu, Jan 18, 2018 at 11:37 PM Mike Sam <[email protected]> wrote: > I am interested to implement an arrow based persisted cache store and I > have a few related questions: > > 1. > > Is it possible just to use Plasma for this goal? > (My understanding is that it is not persistable) > Else, what is the recommended way to do so? > 2. > > Is feather the better file format for persistence to avoid > re-transcoding hot chunks? > 3. > > When Pandas load data from plasma/arrow, is it doubling the memory > usage? (One for the arrow representation, one for pandas representation) > > -- > Thanks, > Mike >
