[
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351230#comment-17351230
]
Sagar Rao commented on KAFKA-9168:
----------------------------------
[~ableegoldman], so I went through the github PR link- which honestly i hadn't
done so far :D - and looks like it enables JNI direct byte buffer for all basic
operations barring transactions. There is a code snippet explaining put
operation using this new option:
{code:java}
// code placeholder
try (RocksDB db = RocksDB.open(opt, "PerformanceTest");
WriteOptions writeOptions = new WriteOptions()) {
writeOptions.setDisableWAL(true);
ByteBuffer directKeyBuffer = ByteBuffer.allocateDirect(128);
directKeyBuffer.order(ByteOrder.BIG_ENDIAN);
ByteBuffer directValueBuffer = ByteBuffer.allocateDirect(128);
directValueBuffer.order(ByteOrder.BIG_ENDIAN);
for (int i = 0; i < 1_000_000; i++) {
directKeyBuffer.clear();
directValueBuffer.clear();
for (int o = 0; o < 16; o++) {
directKeyBuffer.putLong(i);
directValueBuffer.putLong(i);
}
directKeyBuffer.flip();
directValueBuffer.flip();
db.put(writeOptions, directKeyBuffer, directValueBuffer);
}
{code}
As per the benchmarks released on the link, the iterator performace increased
by 37% with 0 GC cycles compared to 293 for the byte array based approach. This
is because there is no referenced memory.
So, looking at these numbers, and to answer to your question about where this
ticket fits in, maybe we can start with some benchmarking the APIs by changing
the way rocksdb state store apis are implemented using this new way? What I
mean is, today if put() API implementation uses the byte[] based APIs, then we
can benchmark using the ByteBuffer based approach and compare the numbers. Do
you think that makes sense?
> Integrate JNI direct buffer support to RocksDBStore
> ---------------------------------------------------
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Sagar Rao
> Assignee: Sagar Rao
> Priority: Major
> Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct
> ByteBuffers in Java. We can look at integrating it whenever it gets merged.
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
