Hi Reynold, Just wanted to ask a off-heap memory general question since we're on this topic, do you guys have a way to do memory management of off-heap memory?
I'm also working on Apache Drill and we use Netty's ByteBuffer abstraction that provides some reference counting mechanisms, but we don't see a good way to manage and detect how leaks happened. Tim On Tue, Aug 27, 2013 at 11:53 AM, Reynold Xin <[email protected]> wrote: > On Tue, Aug 27, 2013 at 1:37 AM, Imran Rashid <[email protected]> > wrote: > > > > > Reynold Xin wrote: > > > This is especially attractive if the application can read directly from > > a byte > > > buffer without generic serialization (like Shark). > > > > interesting -- can you explain how this works in Shark? do you have > > some general way of storing data in byte buffers that avoids > > serialization? Or do you mean that if the user is effectively > > creating an RDD of ints, that you create a an RDD[ByteBuffer], and > > then you read / write ints into the byte buffer yourself? > > Sorry, I'm familiar with the basic idea of shark but not the code at > > all -- even a pointer to the code would be helpful. > > > > > Yes - the user application (in this case) can create a bunch of byte > buffers and doing primitive operations on that directly. >
