anecdotal evidence[1] seems to show that it is well worth while (6x improvement over *nix sockets). i'll be implementing both approaches (sockets and mmap) since that is required for remote operation anyway so we should have some concrete numbers.
best -david [1] http://psy-lob-saw.blogspot.com/2013/04/lock-free-ipc-queue.html On Apr 20, 2013, at 7:48 PM, Ted Dunning <[email protected]> wrote: > I have experimented with this and had decent results, but have not measured > performance. > > One of the real gotchas about Java use of mmap is that you can't easily > unmap a region without JNI help. > > It is worth testing whether you can actually exceed the performance of a > Unix domain socket. Obviously you have some advantages relative to > serialization with mmap, but there are effectively still copies going on > due to NUMA. > > > On Sat, Apr 20, 2013 at 4:21 PM, David Alves <[email protected]> wrote: > >> Thanks for the quick reply and for the pointers. >> wrt to question one, now thinking about it that opens the door for some >> cool optimizations. >> >> wrt to question two I found a couple of interesting references using >> MMAP'd tmpfs, was now looking into the way JCuda uses pinned memory. >> i'll also look into the Peter Lawrey references. >> >> best >> david >> >> On Apr 20, 2013, at 6:00 PM, Jacques Nadeau <[email protected]> wrote: >> >>> On Sat, Apr 20, 2013 at 2:39 PM, David Alves <[email protected]> >> wrote: >>> >>>> Hi >>>> >>>> I'm porting the region level HBase SE to the new SE iface and I >>>> have a couple of questions. >>>> 1- about the method: public ListMultimap<ReadEntry, >>>> DrillbitEndpoint> getReadLocations(Collection<ReadEntry> entries) >>>> >>>> when does it happen that a read entry gets assigned more that one >>>> drillbits? >>>> in terms of hbase I can see the case where multiple read entries >>>> get assigned to the same drillbit (co-located regions) but I can't >> envision >>>> a case where the same read entry (usually corresponding to a shard or >>>> partition) gets assigned to multiple drillbits. when can that happen? >>>> >>> >>> Best example is probably block replica locations in HDFS have multiple >>> possible endpoints. >>> >>> >>> >>>> >>>> 2- with regard to off-heap storage and underlying SE co-location >>>> >>>> this is not really a doubt, just checking that my reasoning is >>>> correct before. >>>> >>>> for co-located underlying SE and Drillbit's we should use >>>> off-heap, shared memory for IPC when possible, correct? >>>> Specifically I'm investigating the possibility of having HBase >>>> store region scan data directly off heap and making the results from >> hbase >>>> contain a set references to aligned shared memory locations. >>>> I'm not sure I'll be implementing this immediately but I'd like >> to >>>> design accounting for it if that is the idea. >>>> Also this means that SE's must work in two modes: co-located with >>>> shared memory and remote with sockets. We'd then have the >>>> Jacques: I'm sure you've put some thought to the underlying >>>> mechanics on how to accomplish this, could you share some quick >>>> ideas/references? >>>> >>> >>> The challenge is separate JVMs don't have a nice way to share memory. >> The >>> simplest way is probably using MMAP'd tmpfs. We'd have to evaluate the >>> performance impact of this complexity. I think the Java Chronicle, >>> HugeCollections or VanillaJava stuff by Peter Lawrey has played with >> this. >>> There isn't a lot of work in the space. Other interesting info: >>> >> http://javaforu.blogspot.com/2011/09/offloading-data-from-jvm-heap-little.html >> . >>> >>> >>> Yes, this does mean that an SE may need to use two different mechanisms >> to >>> interact: one local and one remote/fallback. >>> >>> J >> >>
