[
https://issues.apache.org/jira/browse/HDDS-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong updated HDDS-10382:
-------------------------
Description:
Add "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" to
the ozone script to allow Netty unsafe memory allocation.
According to Netty [doc|https://netty.io/wiki/using-as-a-generic-library.html].
{quote}When a new {{java.nio.ByteBuffer}} is allocated, its content is filled
with zeroes. This "zeroing" consumes CPU cycles and memory bandwidth. Normally,
the buffer is then immediately filled from some data source, so the zeroing did
no good.
{quote}
And Netty has an optimization to avoid this zero assignment, its enablement is
controlled by a flag: PlatformDependent.USE_DIRECT_BUFFER_NO_CLEANER. When this
is enabled netty does the following to allocate a DirectByteBuffer:
1. Call "Unsafe.allocateMemory" to allocate the direct memory. This returns an
address of the allocated buffer.
2. Use reflection to call a private constructor "DirectByteBuffer(long addr,
int cap)" to create the DirectByteBuffer objects. This avoids zeroing the
created buffer, as opposed to the public path "DirectByteBuffer(int cap)".
However, the flag USE_DIRECT_BUFFER_NO_CLEANER is disabled by default in Java
9+ because the reflection in step #2 is disabled by default in Java 9+. This is
controlled by a flag "io.netty.tryReflectionSetAccessible".
The result is today, the zeroing takes a large portion of Netty allocation when
Ozone runs on Java 9+.
!netty_bytebuf_allocation.png|width=695,height=177!
Setting the flag
"-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" helps
Netty leverage the Unsafe allocation and avoid the zeroing cost.
Setting "io.netty.tryReflectionSetAccessible" is a common practice in other
distributed systems, such as HBase, Spark... see:
[https://github.com/search?q=io.netty.tryReflectionSetAccessible&type=code]
A good side-effect of using Netty unsafe allocation is that the used direct
memory is exposed as a metric, see HDDS-9070.
was:
Add "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" to
the ozone script to allow Netty unsafe memory allocation.
According to Netty [doc|https://netty.io/wiki/using-as-a-generic-library.html].
{quote}When a new {{java.nio.ByteBuffer}} is allocated, its content is filled
with zeroes. This "zeroing" consumes CPU cycles and memory bandwidth. Normally,
the buffer is then immediately filled from some data source, so the zeroing did
no good.
{quote}
And Netty has an optimization to avoid this zero assignment, its enablement is
controlled by a flag: PlatformDependent.USE_DIRECT_BUFFER_NO_CLEANER. When this
is enabled netty does the following to allocate a DirectByteBuffer:
1. Call "Unsafe.allocateMemory" to allocate the direct memory. This returns an
address of the allocated buffer.
2. Use reflection to call a private constructor "DirectByteBuffer(long addr,
int cap)" to create the DirectByteBuffer objects. This avoids zeroing the
created buffer, as opposed to the public path "DirectByteBuffer(int cap)".
However, the flag USE_DIRECT_BUFFER_NO_CLEANER is disabled by default in Java
9+ because the reflection in step #2 is disabled by default in Java 9+. This is
controlled by a flag "io.netty.tryReflectionSetAccessible".
The result is today, the zeroing takes a large portion of Netty allocation when
Ozone runs on Java 9+.
!netty_bytebuf_allocation.png|width=695,height=177!
Setting the flag
"-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" helps
Netty leverage the Unsafe allocation and avoid the zeroing cost.
Setting "io.netty.tryReflectionSetAccessible" is a common practice in other
distributed systems, such as HBase, Spark... see:
[https://github.com/search?q=io.netty.tryReflectionSetAccessible&type=code]
> Optimize Netty memory allocation by avoiding zero assignment in Java 9+
> -----------------------------------------------------------------------
>
> Key: HDDS-10382
> URL: https://issues.apache.org/jira/browse/HDDS-10382
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Duong
> Priority: Major
> Attachments: netty_bytebuf_allocation.png
>
>
> Add "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true"
> to the ozone script to allow Netty unsafe memory allocation.
> According to Netty
> [doc|https://netty.io/wiki/using-as-a-generic-library.html].
> {quote}When a new {{java.nio.ByteBuffer}} is allocated, its content is filled
> with zeroes. This "zeroing" consumes CPU cycles and memory bandwidth.
> Normally, the buffer is then immediately filled from some data source, so the
> zeroing did no good.
> {quote}
> And Netty has an optimization to avoid this zero assignment, its enablement
> is controlled by a flag: PlatformDependent.USE_DIRECT_BUFFER_NO_CLEANER. When
> this is enabled netty does the following to allocate a DirectByteBuffer:
> 1. Call "Unsafe.allocateMemory" to allocate the direct memory. This returns
> an address of the allocated buffer.
> 2. Use reflection to call a private constructor "DirectByteBuffer(long addr,
> int cap)" to create the DirectByteBuffer objects. This avoids zeroing the
> created buffer, as opposed to the public path "DirectByteBuffer(int cap)".
> However, the flag USE_DIRECT_BUFFER_NO_CLEANER is disabled by default in Java
> 9+ because the reflection in step #2 is disabled by default in Java 9+. This
> is controlled by a flag "io.netty.tryReflectionSetAccessible".
> The result is today, the zeroing takes a large portion of Netty allocation
> when Ozone runs on Java 9+.
> !netty_bytebuf_allocation.png|width=695,height=177!
> Setting the flag
> "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true"
> helps Netty leverage the Unsafe allocation and avoid the zeroing cost.
> Setting "io.netty.tryReflectionSetAccessible" is a common practice in other
> distributed systems, such as HBase, Spark... see:
> [https://github.com/search?q=io.netty.tryReflectionSetAccessible&type=code]
> A good side-effect of using Netty unsafe allocation is that the used direct
> memory is exposed as a metric, see HDDS-9070.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
