[
https://issues.apache.org/jira/browse/HDDS-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17351:
----------------------------------
Labels: pull-request-available (was: )
> Avoid starting the Hadoop metrics system in short-lived Ozone client processes
> ------------------------------------------------------------------------------
>
> Key: HDDS-17351
> URL: https://issues.apache.org/jira/browse/HDDS-17351
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> *Problem*
> The first datanode client an Ozone client process creates calls
> `XceiverClientManager.getXceiverClientMetrics()`,
> which calls `XceiverClientMetrics.create()`, which runs
> `DefaultMetricsSystem.initialize("XceiverClientMetrics")`.
> That starts a full `MetricsSystemImpl` and registers JMX MBeans for the
> system and each source:
> - it looks for `hadoop-metrics2-xceiverclientmetrics.properties` and
> `hadoop-metrics2.properties`;
> - it builds the metrics config and starts its timer;
> - it registers `XceiverClientMetrics`, and in the `XceiverClientManager`
> constructor `CacheMetrics` too (and
> `ContainerClientMetrics` on writes).
> The client never reads these metrics itself and never sends them to OM or
> SCM. They're only useful where something
> collects them: a metrics sink configured for the client process (the reason
> HDDS-1721 added the `initialize`
> call), or the `/jmx` and Prometheus endpoints of a long-running service that
> embeds the client (S3 Gateway, HttpFS,
> Recon, datanode EC reconstruction).
> Short-lived clients pay the start-up cost and nobody ever reads the result.
> That covers every YARN
> `ContainerLocalizer` JVM, MapReduce task, CLI command and job submitter (eg.
> `hadoop jar`, `spark-submit`). In a fresh JVM on a secure compose
> cluster (Kerberos, gRPC TLS), timed with step markers and DEBUG logs:
> | step of a cold client | total | of which metrics2 |
> |---|---|---|
> | first read (first datanode client) | ~480 ms | **~100 ms**:
> `MetricsSystemImpl` init, `MetricsConfig` file lookups, `MBeans` registration
> |
> | client creation (`FileSystem.get`) | ~600 ms | tens of ms: `CacheMetrics`
> registration |
> A MapReduce job on Ozone starts about four cold clients one after another on
> its critical path: the AppMaster's
> localizer, the AppMaster, a map container's localizer and the map task. So
> this is about 0.4–0.6 s of every job's
> fixed cost.
> *Proposed change*
> 1. **Make registration conditional.** Add a client setting, eg.
> `ozone.client.metrics.mode` in `OzoneClientConfig`:
> - `ALWAYS` (default): today's behaviour.
> - `AUTO`: start the metrics system and register sources only if (a) the
> process has already initialized
> `DefaultMetricsSystem`, which is the case when the client runs inside an
> Ozone service or another
> instrumented service; or (b) a sink is configured for the client's metrics
> prefix. For (b), check for a
> `hadoop-metrics2[-<prefix>].properties` resource defining `*.sink.*` or
> `<prefix>.sink.*` keys, which is
> cheap and doesn't start metrics2.
> - `NEVER`: never register.
> 2. **Keep the metrics objects either way.** `XceiverClientMetrics`,
> `CacheMetrics` and `ContainerClientMetrics` are
> created as plain, unregistered metric objects when registration is off. The
> code paths that increment them, and
> the tests that read them, stay unchanged, and no `MetricsSystemImpl`, MBean
> or timer is created. Registration,
> when it happens, should be done once per process, not per
> `XceiverClientManager` (HDDS-2284 noted the current
> duplicate creation).
> 3. **Default.** Keep `ALWAYS` in this change. Whether `AUTO` can become the
> default later is for discussion: it
> would preserve HDDS-1721's sink case and the embedded-service case, and only
> lose ad-hoc JMX browsing of a
> short-lived client JVM.
> *Compatibility*
> There is no change with the default. Services that embed the client keep
> their client metrics under `AUTO` too,
> because they initialize `DefaultMetricsSystem` before creating clients. Only
> someone who switches to `AUTO` and still wants to
> browse a short-lived client JVM over JMX without a configured sink would lose
> its client metrics; they keep `ALWAYS`.
> *Tests*
> - Unit tests per mode: whether `DefaultMetricsSystem` is initialized and the
> sources are registered, and that the
> counters still work when registration is off.
> - An `AUTO` test with a test-only `hadoop-metrics2` file that configures a
> sink: metrics are registered.
> - An `AUTO` test where the service initialized the metrics system first:
> metrics are registered.
> - Optional: the cold-client probe (create client, getFileStatus, read 64 KB,
> close) shows the ~100 ms drop in
> first-read time under `NEVER`/`AUTO`.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
