Re: RFC: Experiment in accessing/managing persistent memory from Java

[Jonathan Halliday]

Hi Paul

Looks like we're all on the same page regarding the basic approach of 
using a small API and making the critical bits intrinsic. We perhaps 
have some way to go on exactly what that API looks like in terms of the 
classes and methods, but iterating on it by discussion of a JEP seems 
like the best way forward. The important thing from my perspective is 
that so far nobody has come forward with a use case that is not covered 
by the proposed primitives. So it's a small API, but not too small.

As far a tweaks go, we have considered making the low level primitive 
method / intrinsic just a flushCacheline(base_address), since the 
arithmetic and loop for writing flush(from,to) in terms of that low 
level op is something that can be optimized fine by the JIT already. 
Though that does mean exposing the cache line size to the Java layer, 
whereas currently it's only visible in the C code.


My own background and focus is transactions systems, so I'm more about 
the speed and fault tolerance than the capacity, but I can see long vs. 
int being of interest to our Infinispan data grid team and likewise for 
e.g. Oracle Coherence or databases like Cassandra. OTOH it's not 
uncommon to prefer moderately sized files and shard over them, which 
sidesteps the issue.

Utility code to assist with fine-grained memory management within the 
buffer/file may be more useful than support for really large buffers, 
since they tend to be used with some form of internal block/heap 
structure anyhow, rather than to hold very large objects. Providing that 
may be the role of a 3rd party pure Java library like PCJ though, rather 
than something we want in the JDK itself at this early stage. The 
researcher in me is kinda interested in how much of the memory 
allocation and GC code can be re-purposed here though...

What's the intended timeline on long buffer indexing at present? My 
feeling is a pmem API JEP is probably targeting around JDK 13, but we're 
flexible on that.  We may also want to look at related enhancements like 
unmapping buffers. I think those pieces are sufficient decoupled that 
they won't be dependencies for the pmem API though, unlike other factors 
such as the availability of test hardware.

Regards

Jonathan.


On 08/06/18 01:42, Paul Sandoz wrote:
Hi Andrew, Jonathan,

Sandhya gave an overview to a few of us Oracle folks. I agree with what Sandhya 
says regarding the API, a small surface, and on pursuing an unsafe intrinsic. I 
like it and would encourage the writing of a draft JEP, especially to give this 
visibility.

I expect this will be beneficial for experimentation with the Panama foreign 
API where we can use a Pointer to reference into a byte buffer and scribble on 
it. Further, i hope this work may also benefit the persistent collections 
effort (PCJ).


It intersects with https://bugs.openjdk.java.net/browse/JDK-8153111 ((bf) 
Allocating ByteBuffer on heterogeneous memory), which is attempting to be more 
generic.

We might also need to increase the velocity on 
https://bugs.openjdk.java.net/browse/JDK-8180628 (retrofit direct buffer 
support for size beyond gigabyte scales), and i would be very interested your 
views on this, how you might be currently working around such size limitations, 
and what buffer enhancements would work for you.

Thanks,
Paul.


On May 30, 2018, at 10:21 PM, Viswanathan, Sandhya 
<sandhya.viswanat...@intel.com> wrote:

Hi Andrew/Jonathan,
  
Thanks a lot for sharing this work. Copying hotspot-compiler-dev to get their feedback as well.
  
Couple of thoughts/observations below:
* Supporting ByteBuffer on persistent memory using existing FileChannel and 
MappedByteBuffer mechanism sounds like a very good idea.
  
* Extending FileChannel.map to take additional parameter to indicate that the ByteBuffer is backed by persistent memory is a small API change.
  
* Adding MappedByteBuffer.force(int from, int to) method on smaller range is very useful in addition to the force() on entire ByteBuffer.
  
* The underlying force0_mapsync() could be implemented in terms of new unsafe APIs which in turn could be intrinsified.
    The advantage of this is that the unsafe APIs could then be used for other 
future persistent memory APIs in the JRE.
    Specifically the following two unsafe APIs would be useful:
    a) public native void flush(long address, long size);
    b) public native void storeFence();
    storeFence() exists today but doesn’t generate any instruction for x86.
    Wondering if we could have additional boolean parameter to force the sfence 
generation.
  * DEFAULT_CACHE_LINE_SIZE is 128 in 
src/hotspot/cpu/x86/globalDefinitions_x86.hpp whereas actual cache line on the 
hardware is 64 bytes.
     This could be the cause for some of performance that you saw with compiler 
intrinsic vs pure C native.
  
Best Regards,
Sandhya
  
  
  
RFC: Experiment in accessing/managing persistent memory from Java
Andrew Dinn adinn at redhat.com
Mon May 21 09:47:46 UTC 2018
  
I have been helping one of my Red Hat colleagues, Jonathan Halliday, to
investigate provision of a Java equivalent to Intel's libpmem suite of C
libraries [1]. This approach avoids the significant cost of using the
Intel libraries from Java via JNI (or, worse, as a virtual driver for a
persistent memory device).
  
   Jonathan has modified the JVM/JDK to allow a MappedByteBuffer to be
mapped over persistent memory, providing equivalent function to libpmem
itself.
  
   On top of this he implemented a Java journaled log class providing
equivalent functionality to one of the Intel client libs, libpmemlog,
built over libpmem.
  
   The modified MappedByteBuffer can be configured to use either i) a
registered native method or ii) a JIT intrinsic to perform the critical
task of cache line writeback i.e. the persistence step (the intrinsic is
my contribution).
  
Jonathan's tests compare use of JNI, registered native and intrinsic
with an equivalent C program to write a large swathe of records to a
journaled log file stored in persistent memory. Performance is worse
than C when relying on JNI and significantly better with JVM/JDK
support. Indeed, as one might reasonably expect, use of the JIT
intrinsic almost completely eliminates writeback costs.
  
The journaled log code, jdk dev tree patch, build instructions, test
code plus C equivalent and test results are all available from
Jonathan's git repo [2].
  
For those who do not want to look at the actual code, the README file
[3] provides background to use of persistent memory, an overview of the
design, and summary details of the test process and results.
  
[1] https://pmem.io/pmdk/
[2] https://github.com/jhalliday/pmem
[3] http://github.com://jhalliday/pmem/README.md
  
n.b. Jonathan has experimented with using this same prototype to replace
the journaled log used in the Red Hat Narayana transaction manager. It
provides a significant improvement on the current disk file based log,
both for throughput and latency (the code is not yet available as
getting it to work involved some horrible hacking of the build to
migrate up to jdk11).
  
  
regards,
  
  
Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander


--
Registered in England and Wales under Company Registration No. 03798903 
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander