Andrey Yarovoy created HDDS-17357:
-------------------------------------
Summary: Initialize Netty and netty-tcnative in the background
when an Ozone client is created
Key: HDDS-17357
URL: https://issues.apache.org/jira/browse/HDDS-17357
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
*Problem*
A cold Ozone client initializes Netty, and in secure clusters also
netty-tcnative (BoringSSL), on the main thread at
its first datanode connection. That is `NettyChannelBuilder` /
`GrpcSslContexts.forClient()` in
`XceiverClientGrpc.connectToDatanode()`, or the Ratis gRPC client for writes.
Nothing else happens in parallel:
the read or write waits for it.
On a secure compose cluster (Kerberos + gRPC TLS, JDK 25), with DEBUG logs, the
time from "Connecting to server" to
netty-tcnative being ready is **about 115 ms** (112–120 ms, 5 runs):
- about half is Netty itself: `PlatformDependent`, the native epoll transport,
the event loop and the pooled buffer
allocator;
- about 80 ms is netty-tcnative:
- roughly half of that is Java class loading;
- the rest is fixed native work in `OpenSsl`'s static initializer: test SSL
contexts, the cipher list, protocol
and OCSP probes, and a test certificate and key.
The cost is hard to cut directly, measured in fresh JVMs:
- copying the library out of the jar is negligible: a tmpfs work dir saves 0
ms, a pre-extracted library on
`java.library.path` 8 ms;
- `OpenSsl` has no lazy path and no setting that skips its probes.
But the work doesn't depend on anything the client does first, and the client
knows early that it will need it. Its
first datanode connection comes more than 0.6 s after creation, after the
Kerberos login and the OM calls.
The cost shows up wherever many short-lived client JVMs start: YARN
`ContainerLocalizer` JVMs, MapReduce tasks, CLI
commands. A MapReduce job starts about four cold clients one after another on
its critical path.
*Proposed change*
When an `XceiverClientManager` is created (in `XceiverClientCreator`'s
constructor), start a one-shot daemon thread,
at most once per JVM, that:
1. initializes Netty: `PlatformDependent`, and allocates and releases one
buffer from
`PooledByteBufAllocator.DEFAULT`;
2. when security and gRPC TLS are enabled (`SecurityConfig.isSecurityEnabled()
&& isGrpcTlsEnabled()`), also calls
`OpenSsl.isAvailable()`, which loads netty-tcnative and runs its static
initializer.
Errors in the thread are logged at DEBUG and otherwise ignored. The connection
path stays unchanged: it uses
whatever is already initialized, or does it itself as today. A setting to turn
the warm-up off (default on) is
optional.
*Results*
The cold-client probe on a secure compose cluster: a fresh JVM creates the
`ofs://` client, calls getFileStatus,
reads 64 KB and closes. The warm-up runs on a thread started right after
loading the configuration, as the proposed
change would. Ten interleaved rounds, medians:
| client | create the client | read | total (JVM start to end) |
|---|---|---|---|
| as today | 757 ms | 575 ms | 1,714 ms |
| Netty warmed up in the background | 754 ms | 523 ms | 1,650 ms (**−64 ms**) |
| Netty and netty-tcnative warmed up | 724 ms | 480 ms | 1,578 ms (**−136 ms**)
|
The totals' ranges don't overlap: 1,527–1,608 ms with the full warm-up,
1,669–1,745 ms as today. The saving exceeds
the ~115 ms step, because `OpenSsl`'s probe also loads the JDK X.509 classes
that the client uses to parse the CA
certificates. Across a job's four cold clients that's about 0.5 s.
*Compatibility and risks*
- **No change in behavior or configuration:** the same classes are initialized,
just earlier and on another thread.
`GrpcSslContexts.forClient()` already picks OpenSSL whenever it's available, so
warming it doesn't change which
provider is used.
- **CPU:** the gain needs a spare core during client creation. In a container
capped at one core, the thread
competes with the main thread and the gain shrinks, but there's no loss beyond
the thread itself.
- **Class initialization:** if the main thread reaches a Netty class while the
warm-up thread is still initializing
it, the main thread waits for it, which costs no more than doing it itself. The
warm-up thread holds no Ozone
locks, so it can't deadlock with the client.
- **Long-lived processes** (S3 Gateway, services that create clients): a
one-time cost they would pay anyway.
*Tests*
- Unit: creating an `XceiverClientManager` starts the warm-up at most once per
JVM. It starts the TLS part only when
security and gRPC TLS are both enabled. A failure in the warm-up doesn't fail
client creation.
- Existing gRPC and Ratis client tests, with and without TLS, pass unchanged.
- Measurement: the cold-client probe on a secure compose cluster shows the
total about 130 ms shorter.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
