[
https://issues.apache.org/jira/browse/ARROW-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445095#comment-17445095
]
Benjamin Wilhelm commented on ARROW-11901:
------------------------------------------
I will just add one small comment for now:
{quote}But the
[API|https://github.com/bytedeco/javacpp-presets/pull/1094/files#diff-3d9af736e997982d68098d986670f05ff40ae0cc62773a1dd0eb418e55990317R38]
isn't quite what I imagined, it looks like it goes through ByteBuffer, when
all we really need is something like [ZSTD
API|https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/Zstd.java#L454]
{quote}
I just used {{ByteBuffer}} for the example code. If the memory address is known
(which it is for {{{}ArrowBuf{}}}) a Pointer to this memory address can be used
(which I do in my implementation of {{{}CompressionCodec{}}}). The API is still
a bit annoying to use (mainly because decompression cannot be done with one
call and it does not feel like a Java API) but it is not limited to
{{Integer.MAX_VALUE}} bytes. Maybe, there could be a convenience layer in the
{{org.bytedeco.lz4}} lib.
> [Java] Investigate potential performance improvement of compression codec
> -------------------------------------------------------------------------
>
> Key: ARROW-11901
> URL: https://issues.apache.org/jira/browse/ARROW-11901
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Java
> Reporter: Liya Fan
> Priority: Major
>
> In response to the discussion in
> https://github.com/apache/arrow/pull/8949/files#r588046787
> There are some performance penalties in the implementation of the compression
> codecs (e.g. data copying between heap/off-heap data). We need to revise the
> code to improve the performance.
> We should also provide some benchmarks to validate that the performance
> actually improves.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
