[
https://issues.apache.org/jira/browse/HDDS-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17357:
----------------------------------
Labels: pull-request-available (was: )
> Initialize Netty and netty-tcnative in the background when an Ozone client is
> created
> -------------------------------------------------------------------------------------
>
> Key: HDDS-17357
> URL: https://issues.apache.org/jira/browse/HDDS-17357
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> *Problem*
> A cold Ozone client initializes Netty, and in secure clusters also
> netty-tcnative (BoringSSL), on the main thread at
> its first datanode connection. That is `NettyChannelBuilder` /
> `GrpcSslContexts.forClient()` in
> `XceiverClientGrpc.connectToDatanode()`, or the Ratis gRPC client for writes.
> Nothing else happens in parallel:
> the read or write waits for it.
> On a secure compose cluster (Kerberos + gRPC TLS, JDK 25), with DEBUG logs,
> the time from "Connecting to server" to
> netty-tcnative being ready is **about 115 ms** (112–120 ms, 5 runs):
> - about half is Netty itself: `PlatformDependent`, the native epoll
> transport, the event loop and the pooled buffer
> allocator;
> - about 80 ms is netty-tcnative:
> - roughly half of that is Java class loading;
> - the rest is fixed native work in `OpenSsl`'s static initializer: test SSL
> contexts, the cipher list, protocol
> and OCSP probes, and a test certificate and key.
> The cost is hard to cut directly, measured in fresh JVMs:
> - copying the library out of the jar is negligible: a tmpfs work dir saves 0
> ms, a pre-extracted library on
> `java.library.path` 8 ms;
> - `OpenSsl` has no lazy path and no setting that skips its probes.
> But the work doesn't depend on anything the client does first, and the client
> knows early that it will need it. Its
> first datanode connection comes more than 0.6 s after creation, after the
> Kerberos login and the OM calls.
> The cost shows up wherever many short-lived client JVMs start: YARN
> `ContainerLocalizer` JVMs, MapReduce tasks, CLI
> commands. A MapReduce job starts about four cold clients one after another on
> its critical path.
> *Proposed change*
> When an `XceiverClientManager` is created (in `XceiverClientCreator`'s
> constructor), start a one-shot daemon thread,
> at most once per JVM, that:
> 1. initializes Netty: `PlatformDependent`, and allocates and releases one
> buffer from
> `PooledByteBufAllocator.DEFAULT`;
> 2. when security and gRPC TLS are enabled
> (`SecurityConfig.isSecurityEnabled() && isGrpcTlsEnabled()`), also calls
> `OpenSsl.isAvailable()`, which loads netty-tcnative and runs its static
> initializer.
> Errors in the thread are logged at DEBUG and otherwise ignored. The
> connection path stays unchanged: it uses
> whatever is already initialized, or does it itself as today. A setting to
> turn the warm-up off (default on) is
> optional.
> *Results*
> The cold-client probe on a secure compose cluster: a fresh JVM creates the
> `ofs://` client, calls getFileStatus,
> reads 64 KB and closes. The warm-up runs on a thread started right after
> loading the configuration, as the proposed
> change would. Ten interleaved rounds, medians:
> | client | create the client | read | total (JVM start to end) |
> |---|---|---|---|
> | as today | 757 ms | 575 ms | 1,714 ms |
> | Netty warmed up in the background | 754 ms | 523 ms | 1,650 ms (**−64 ms**)
> |
> | Netty and netty-tcnative warmed up | 724 ms | 480 ms | 1,578 ms (**−136
> ms**) |
> The totals' ranges don't overlap: 1,527–1,608 ms with the full warm-up,
> 1,669–1,745 ms as today. The saving exceeds
> the ~115 ms step, because `OpenSsl`'s probe also loads the JDK X.509 classes
> that the client uses to parse the CA
> certificates. Across a job's four cold clients that's about 0.5 s.
> *Compatibility and risks*
> - **No change in behavior or configuration:** the same classes are
> initialized, just earlier and on another thread.
> `GrpcSslContexts.forClient()` already picks OpenSSL whenever it's available,
> so warming it doesn't change which
> provider is used.
> - **CPU:** the gain needs a spare core during client creation. In a container
> capped at one core, the thread
> competes with the main thread and the gain shrinks, but there's no loss
> beyond the thread itself.
> - **Class initialization:** if the main thread reaches a Netty class while
> the warm-up thread is still initializing
> it, the main thread waits for it, which costs no more than doing it itself.
> The warm-up thread holds no Ozone
> locks, so it can't deadlock with the client.
> - **Long-lived processes** (S3 Gateway, services that create clients): a
> one-time cost they would pay anyway.
> *Tests*
> - Unit: creating an `XceiverClientManager` starts the warm-up at most once
> per JVM. It starts the TLS part only when
> security and gRPC TLS are both enabled. A failure in the warm-up doesn't fail
> client creation.
> - Existing gRPC and Ratis client tests, with and without TLS, pass unchanged.
> - Measurement: the cold-client probe on a secure compose cluster shows the
> total about 130 ms shorter.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
