I was at a Meetup last night that was talking about how Spark does this natively in some cases with sun.misc.unsafe (which is being removed in Java 11) and how Flink does this with DirectByteBuffer (I think?) which has numerous benefits (that's what the meetup talk was about).
On Sun, Nov 3, 2019 at 8:19 PM Andrew Palumbo (via Google Docs) < andrewpalumbo2...@gmail.com> wrote: > I've shared an item with you: > > Proposal for a New Serialization, De-serialization and Memory Scheme for > [Spark] distribution of Mahout AbstractVectors and Mahout Abstract > Matrices: Direct to Main (off heap), JVM (on heap), GPU and other devices > > https://docs.google.com/document/d/18RybVEpjqjDU_cCzwM6dtS3ZZkd1tDvCYlkhqzloS-4/edit?usp=sharing&ts=5dbf8960 > > It's not an attachment -- it's stored online. To open this item, just > click > the link above. >
