tannnmayy opened a new pull request, #40986: URL: https://github.com/apache/beam/pull/40986
Fixes #23142 The [RunInference metrics](https://beam.apache.org/documentation/ml/runinference-metrics/) page doesn't define all the metrics, and two of its statements are wrong: - It says `inference_request_batch_size_COUNT` is the total number of samples. The distribution gets one value per batch (`_MetricsCollector.update()` in `base.py` is called once per batch), so its COUNT is the number of batches. The screenshot on the page shows this: `inference_request_batch_size_COUNT` is 4,013, but `num_inferences` is 6,000. - It gives the total inference time as `num_inferences x inference_batch_latency_micro_secs_MEAN`. That multiplies a number of elements by an average time per batch, so it overstates the time by a factor of the average batch size. With the screenshot's numbers it gives about 4 h 18 min, longer than the 2 h 52 min 44 s the Inference step ran, while `inference_batch_latency_micro_secs_COUNT x _MEAN` gives about 2 h 52 min. - `failed_batches_counter` isn't mentioned. This PR: - replaces the bullet list with a table of all seven metrics (the six from the issue and `failed_batches_counter`) and their type; - explains how to read the distributions: the Cloud console shows each one as `_COUNT`, `_MIN`, `_MAX` and `_MEAN`, `_COUNT` is the number of recorded values (one per successful batch, or one per model load), and failed batches aren't recorded; - notes that `model_byte_size` is the growth of the worker process's peak memory while the model loads, and that it's always 0 where the `resource` module isn't available (Windows); - says which namespace the metrics are reported in: the model handler's `get_metrics_namespace()` (for example `BeamML_PyTorch`), or `metrics_namespace` if it's passed to `RunInference`; - fixes the total inference time formula. Testing: - Checked each statement against `_MetricsCollector` and `_RunInferenceDoFn` in `base.py`. - Ran small RunInference pipelines on the FnApiRunner with a fake model handler (100 inputs in batches of 10, 50 ms per batch), on Linux (Python 3.12) and Windows (Python 3.11): - `num_inferences` was 100 and the COUNT of each batch distribution was 10; - latency COUNT x MEAN was 0.50 s, while the old formula gave 5.03 s; - with one failing batch and `with_exception_handling()`, `failed_batches_counter` was 1, `num_inferences` was 90, and the batch distributions' COUNT was 9; - a model that allocates 200 MB in `load_model()` reported a `model_byte_size` of 200 MB on Linux and 0 on Windows; - the metrics were reported in the handler's namespace, or in `metrics_namespace` when it was set. - The table uses the same `{{< table >}}` shortcode as the other tables on the ML pages. R: @jrmccluskey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
