On Tue, Aug 27, 2013 at 1:37 AM, Imran Rashid <[email protected]> wrote:
> > Reynold Xin wrote: > > This is especially attractive if the application can read directly from > a byte > > buffer without generic serialization (like Shark). > > interesting -- can you explain how this works in Shark? do you have > some general way of storing data in byte buffers that avoids > serialization? Or do you mean that if the user is effectively > creating an RDD of ints, that you create a an RDD[ByteBuffer], and > then you read / write ints into the byte buffer yourself? > Sorry, I'm familiar with the basic idea of shark but not the code at > all -- even a pointer to the code would be helpful. > > Yes - the user application (in this case) can create a bunch of byte buffers and doing primitive operations on that directly.
