[
https://issues.apache.org/jira/browse/HDDS-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong updated HDDS-10288:
-------------------------
Description:
Ozone client should leverage direct buffers and zero-copy to improve
performance while writing and reading data.
However, today the checksum algorithm doesn't support direct buffer yet,
because Java 8 doesn't support it yet.
Ref:
https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBufferImpl.java#L69-L69
{code:java}
@Override
// TODO - when we eventually move to a minimum Java version >= 9 this method
// should be refactored to simply call checksum.update(buffer), as the
// Checksum interface has been enhanced to allow this since Java 9.
public void update(ByteBuffer buffer) {
....
if (buffer.hasArray()) {
checksum.update(buffer.array(), buffer.position() + buffer.arrayOffset(),
buffer.remaining());
} else {
byte[] b = new byte[buffer.remaining()];
buffer.get(b);
checksum.update(b, 0, b.length);
}
}
{code}
While we accept Java 8 as the minimum supported JVM, most Ozone production
clusters are running on a newer platform like JDK11.
We can leverage Java 9+ interface using reflection to checksum direct buffers.
Refer to this example in Kafka.
https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/Checksums.java
was:
Ozone client should leverage direct buffers and zero-copy to improve
performance while writing and reading data.
However, today the checksum algorithm doesn't support direct buffer yet,
because Java 8 doesn't support it yet.
Ref:
https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBufferImpl.java#L69-L69
{code:java}
@Override
// TODO - when we eventually move to a minimum Java version >= 9 this method
// should be refactored to simply call checksum.update(buffer), as the
// Checksum interface has been enhanced to allow this since Java 9.
public void update(ByteBuffer buffer) {
....
if (buffer.hasArray()) {
checksum.update(buffer.array(), buffer.position() + buffer.arrayOffset(),
buffer.remaining());
} else {
byte[] b = new byte[buffer.remaining()];
buffer.get(b);
checksum.update(b, 0, b.length);
}
}
{code}
While we accept Java 8 as the minimum supported JVM, most of the Ozone
production cluster is running on a newer java like JDK11.
We can leverage Java 9+ interface using reflection to checksum direct buffers.
Refer to this example in Kafka.
https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/Checksums.java
> Checksum to support direct buffers
> ----------------------------------
>
> Key: HDDS-10288
> URL: https://issues.apache.org/jira/browse/HDDS-10288
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client
> Reporter: Duong
> Assignee: Duong
> Priority: Major
>
> Ozone client should leverage direct buffers and zero-copy to improve
> performance while writing and reading data.
> However, today the checksum algorithm doesn't support direct buffer yet,
> because Java 8 doesn't support it yet.
> Ref:
> https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChecksumByteBufferImpl.java#L69-L69
> {code:java}
> @Override
> // TODO - when we eventually move to a minimum Java version >= 9 this method
> // should be refactored to simply call checksum.update(buffer), as
> the
> // Checksum interface has been enhanced to allow this since Java 9.
> public void update(ByteBuffer buffer) {
> ....
> if (buffer.hasArray()) {
> checksum.update(buffer.array(), buffer.position() +
> buffer.arrayOffset(),
> buffer.remaining());
> } else {
> byte[] b = new byte[buffer.remaining()];
> buffer.get(b);
> checksum.update(b, 0, b.length);
> }
> }
> {code}
> While we accept Java 8 as the minimum supported JVM, most Ozone production
> clusters are running on a newer platform like JDK11.
> We can leverage Java 9+ interface using reflection to checksum direct
> buffers.
> Refer to this example in Kafka.
> https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/Checksums.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
