[
https://issues.apache.org/jira/browse/HDFS-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HDFS-14306:
--------------------------------
Description:
Filesystem connectors that don't natively support {{ByteBuffer}} cannot fully
take advantage of the direct read optimizations added in HDFS-2834 / HDFS-3110.
In this case, they fall back to the regular read path which allocates a Java
byte array for every read. This leads to a lot of perf overhead wasted in
{{TypeArrayKlass::allocate_common}}.
A performance optimization would be to re-use Java byte arrays between calls to
{{hdfsRead}} / {{hdfsPread}} so that new arrays don't need to be allocate for
every single call.
was:
Filesystem connectors that don't natively support {{ByteBuffer}}s cannot fully
take advantage of the direct read optimizations added in HDFS-2834 / HDFS-3110.
In this case, they fall back to the regular read path which allocates a Java
byte array for every read. This leads to a lot of perf overhead wasted in
{{TypeArrayKlass::allocate_common}}.
A performance optimization would be to re-use Java byte arrays between calls to
{{hdfsRead}} / {{hdfsPread}} so that new arrays don't need to be allocate for
every single call.
> Re-use Java buffers across libhdfs read calls
> ---------------------------------------------
>
> Key: HDFS-14306
> URL: https://issues.apache.org/jira/browse/HDFS-14306
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> Filesystem connectors that don't natively support {{ByteBuffer}} cannot fully
> take advantage of the direct read optimizations added in HDFS-2834 /
> HDFS-3110. In this case, they fall back to the regular read path which
> allocates a Java byte array for every read. This leads to a lot of perf
> overhead wasted in {{TypeArrayKlass::allocate_common}}.
> A performance optimization would be to re-use Java byte arrays between calls
> to {{hdfsRead}} / {{hdfsPread}} so that new arrays don't need to be allocate
> for every single call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
