[
https://issues.apache.org/jira/browse/HDDS-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-16985:
----------------------------------
Description:
*Problem*
The OM gRPC server (used by S3 Gateway through `GrpcOmTransport`) always uses
Netty's NIO transport, even on Linux where native epoll is available. The
shipped tarball also cannot load unshaded native epoll at all, so the S3
Gateway's OM gRPC client falls back to NIO without any warning.
*Current state*
1. *{*}`GrpcOzoneManagerServer` hard-codes NIO:{*}*
- boss and worker groups are `new NioEventLoopGroup(...)`
-`NettyServerBuilder.channelType(NioServerSocketChannel.class)`
`XceiverServerGrpc` on the datanode already switches on `Epoll.isAvailable()`
and picks `EpollEventLoopGroup` / `EpollServerSocketChannel` when it can. The
OM server has no such switch.
2. *{*}The dist is missing the native library for unshaded Netty.{*}*
`share/ozone/lib` contains only `netty-transport-native-epoll-<ver>.jar` with
no classifier, and that jar has no `.so` in it. The `linux-x86_64` and
`linux-aarch_64` classifier jars are not packaged. As a result,
`io.netty.channel.epoll.Epoll.isAvailable()` returns `false` at runtime, which
affects:
-`GrpcOzoneManagerServer`: it would stay on NIO even after fix 1 unless this is
also fixed.
-`GrpcOmTransport` (S3 Gateway → OM client): it does not set a transport
explicitly, so gRPC chooses epoll only when it is available. Today it always
ends up on NIO.
The shaded gRPC/Netty from ratis-thirdparty is not affected, because
`ratis-thirdparty-misc` bundles the shaded epoll `.so`. So the datanode, SCM HA
and Ratis gRPC paths already get epoll on Linux.
3. *{*}Minor:{*}* the datanode `StreamingServer` / `StreamingClient`
(`container/stream`) hard-code `NioEventLoopGroup(100)` and
`Nio[Server]SocketChannel`. Today they are only called from tests and freon
`StreamingGenerator`.
#
## Proposed change
- In `GrpcOzoneManagerServer`, select `EpollEventLoopGroup` +
`EpollServerSocketChannel` when `Epoll.isAvailable()` is true and keep NIO as
the fallback, mirroring `XceiverServerGrpc`. Log the selected channel type at
startup.
- Package `io.netty:netty-transport-native-epoll` with the `linux-x86_64` and
`linux-aarch_64` classifiers in the dist, and update `jar-report.txt` and
`LICENSE.txt`.
- Optional: apply the same switch to `StreamingServer` / `StreamingClient`.
No protocol, config or compatibility impact. Platforms without epoll keep using
NIO.
*Testing*
- Existing OM gRPC / S3 Gateway unit and integration tests, which use the NIO
fallback on macOS.
- Linux compose cluster: confirm the OM startup log shows the epoll channel
type, and that jstack shows epoll event-loop threads in OM and S3 Gateway.
was:
## Problem
The OM gRPC server (used by S3 Gateway through `GrpcOmTransport`) always uses
Netty's NIO transport, even on Linux where native epoll is available. The
shipped tarball also cannot load unshaded native epoll at all, so the S3
Gateway's OM gRPC client falls back to NIO without any warning.
## Current state
1. **`GrpcOzoneManagerServer` hard-codes NIO:**
- boss and worker groups are `new NioEventLoopGroup(...)`
-`NettyServerBuilder.channelType(NioServerSocketChannel.class)`
`XceiverServerGrpc` on the datanode already switches on `Epoll.isAvailable()`
and picks `EpollEventLoopGroup` / `EpollServerSocketChannel` when it can. The
OM server has no such switch.
2. **The dist is missing the native library for unshaded Netty.**
`share/ozone/lib` contains only `netty-transport-native-epoll-<ver>.jar` with
no classifier, and that jar has no `.so` in it. The `linux-x86_64` and
`linux-aarch_64` classifier jars are not packaged. As a result,
`io.netty.channel.epoll.Epoll.isAvailable()` returns `false` at runtime, which
affects:
-`GrpcOzoneManagerServer`: it would stay on NIO even after fix 1 unless this is
also fixed.
-`GrpcOmTransport` (S3 Gateway → OM client): it does not set a transport
explicitly, so gRPC chooses epoll only when it is available. Today it always
ends up on NIO.
The shaded gRPC/Netty from ratis-thirdparty is not affected, because
`ratis-thirdparty-misc` bundles the shaded epoll `.so`. So the datanode, SCM HA
and Ratis gRPC paths already get epoll on Linux.
3. **Minor:** the datanode `StreamingServer` / `StreamingClient`
(`container/stream`) hard-code `NioEventLoopGroup(100)` and
`Nio[Server]SocketChannel`. Today they are only called from tests and freon
`StreamingGenerator`.
## Proposed change
- In `GrpcOzoneManagerServer`, select `EpollEventLoopGroup` +
`EpollServerSocketChannel` when `Epoll.isAvailable()` is true and keep NIO as
the fallback, mirroring `XceiverServerGrpc`. Log the selected channel type at
startup.
- Package `io.netty:netty-transport-native-epoll` with the `linux-x86_64` and
`linux-aarch_64` classifiers in the dist, and update `jar-report.txt` and
`LICENSE.txt`.
- Optional: apply the same switch to `StreamingServer` / `StreamingClient`.
No protocol, config or compatibility impact. Platforms without epoll keep using
NIO.
## Testing
- Existing OM gRPC / S3 Gateway unit and integration tests, which use the NIO
fallback on macOS.
- Linux compose cluster: confirm the OM startup log shows the epoll channel
type, and that jstack shows epoll event-loop threads in OM and S3 Gateway.
> OM gRPC server / dist do not use Netty native epoll
> ---------------------------------------------------
>
> Key: HDDS-16985
> URL: https://issues.apache.org/jira/browse/HDDS-16985
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> *Problem*
> The OM gRPC server (used by S3 Gateway through `GrpcOmTransport`) always uses
> Netty's NIO transport, even on Linux where native epoll is available. The
> shipped tarball also cannot load unshaded native epoll at all, so the S3
> Gateway's OM gRPC client falls back to NIO without any warning.
> *Current state*
> 1. *{*}`GrpcOzoneManagerServer` hard-codes NIO:{*}*
> - boss and worker groups are `new NioEventLoopGroup(...)`
> -`NettyServerBuilder.channelType(NioServerSocketChannel.class)`
> `XceiverServerGrpc` on the datanode already switches on `Epoll.isAvailable()`
> and picks `EpollEventLoopGroup` / `EpollServerSocketChannel` when it can. The
> OM server has no such switch.
> 2. *{*}The dist is missing the native library for unshaded Netty.{*}*
> `share/ozone/lib` contains only `netty-transport-native-epoll-<ver>.jar` with
> no classifier, and that jar has no `.so` in it. The `linux-x86_64` and
> `linux-aarch_64` classifier jars are not packaged. As a result,
> `io.netty.channel.epoll.Epoll.isAvailable()` returns `false` at runtime,
> which affects:
> -`GrpcOzoneManagerServer`: it would stay on NIO even after fix 1 unless this
> is also fixed.
> -`GrpcOmTransport` (S3 Gateway → OM client): it does not set a transport
> explicitly, so gRPC chooses epoll only when it is available. Today it always
> ends up on NIO.
> The shaded gRPC/Netty from ratis-thirdparty is not affected, because
> `ratis-thirdparty-misc` bundles the shaded epoll `.so`. So the datanode, SCM
> HA and Ratis gRPC paths already get epoll on Linux.
> 3. *{*}Minor:{*}* the datanode `StreamingServer` / `StreamingClient`
> (`container/stream`) hard-code `NioEventLoopGroup(100)` and
> `Nio[Server]SocketChannel`. Today they are only called from tests and freon
> `StreamingGenerator`.
> #
> ## Proposed change
> - In `GrpcOzoneManagerServer`, select `EpollEventLoopGroup` +
> `EpollServerSocketChannel` when `Epoll.isAvailable()` is true and keep NIO as
> the fallback, mirroring `XceiverServerGrpc`. Log the selected channel type at
> startup.
> - Package `io.netty:netty-transport-native-epoll` with the `linux-x86_64`
> and `linux-aarch_64` classifiers in the dist, and update `jar-report.txt` and
> `LICENSE.txt`.
> - Optional: apply the same switch to `StreamingServer` / `StreamingClient`.
> No protocol, config or compatibility impact. Platforms without epoll keep
> using NIO.
> *Testing*
> - Existing OM gRPC / S3 Gateway unit and integration tests, which use the
> NIO fallback on macOS.
> - Linux compose cluster: confirm the OM startup log shows the epoll channel
> type, and that jstack shows epoll event-loop threads in OM and S3 Gateway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
