jrmccluskey commented on code in PR #40986:
URL: https://github.com/apache/beam/pull/40986#discussion_r4589024130


##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -87,20 +87,26 @@ The following image shows a snapshot of different metrics 
in the Google Cloud co
 
   ![RunInference GPU metrics rendered on 
Dataflow](/images/runinference_metrics_snapshot.svg)
 
-Some metrics commonly used for benchmarking are:
+RunInference reports the following metrics:
 
-* `num_inferences`: Represents the total number of elements passed to 
`run_inference()`.
+{{< table >}}
+| Metric | Type | Description |
+| ------ | ---- | ----------- |
+| `num_inferences` | Counter | The total number of elements passed to 
`run_inference()`. Elements in batches where inference failed aren't counted. |
+| `failed_batches_counter` | Counter | The number of batches for which 
inference raised an exception. |
+| `inference_request_batch_size` | Distribution | The number of elements in 
each batch of examples (created from `beam.BatchElements`) passed to 
`run_inference()`. |
+| `inference_request_batch_byte_size` | Distribution | The size of each batch 
of examples, in bytes, as computed by the model handler's `get_num_bytes()` 
method. |
+| `inference_batch_latency_micro_secs` | Distribution | The time taken to 
perform inference on each batch of examples, measured in microseconds. |
+| `load_model_latency_milli_secs` | Distribution | The time taken to load and 
initialize the model, measured in milliseconds. |
+| `model_byte_size` | Distribution | The memory consumed to load and 
initialize the model, in bytes. RunInference measures it as the increase in the 
worker process's peak memory usage while the model loads. On platforms where 
the Python `resource` module isn't available, such as Windows, this metric is 
always 0. |
+{{< /table >}}
 
-* `inference_batch_latency_micro_secs_MEAN`: Represents the average time taken 
to perform inference across all batches of examples, measured in microseconds.
+Counters report a total, and distributions record one value at a time. For 
more information, see [Types of 
metrics](/documentation/programming-guide/#types-of-metrics). In the Google 
Cloud console, each distribution appears as four metrics with the suffixes 
`_COUNT`, `_MIN`, `_MAX`, and `_MEAN`, where `_COUNT` is the number of values 
recorded.
 
-* `inference_request_batch_size_COUNT`: Represents the total number of samples 
across all batches of examples (created from `beam.BatchElements`) to be passed 
to `run_inference()`.
+`inference_request_batch_size`, `inference_request_batch_byte_size`, and 
`inference_batch_latency_micro_secs` record one value for each batch where 
inference succeeded, so their `_COUNT` is the number of batches, not the number 
of elements. For example, in the preceding image, 
`inference_request_batch_size_COUNT` is 4,013, the number of batches, while 
`num_inferences` is 6,000, the number of elements. 
`load_model_latency_milli_secs` and `model_byte_size` record one value each 
time the model is loaded.

Review Comment:
   Elaborating on the distinction between the element and batch counts only 
makes sense in the context of the previous state of the page and isn't 
particularly necessary. 



##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -87,20 +87,26 @@ The following image shows a snapshot of different metrics 
in the Google Cloud co
 
   ![RunInference GPU metrics rendered on 
Dataflow](/images/runinference_metrics_snapshot.svg)
 
-Some metrics commonly used for benchmarking are:
+RunInference reports the following metrics:
 
-* `num_inferences`: Represents the total number of elements passed to 
`run_inference()`.
+{{< table >}}
+| Metric | Type | Description |
+| ------ | ---- | ----------- |
+| `num_inferences` | Counter | The total number of elements passed to 
`run_inference()`. Elements in batches where inference failed aren't counted. |
+| `failed_batches_counter` | Counter | The number of batches for which 
inference raised an exception. |
+| `inference_request_batch_size` | Distribution | The number of elements in 
each batch of examples (created from `beam.BatchElements`) passed to 
`run_inference()`. |
+| `inference_request_batch_byte_size` | Distribution | The size of each batch 
of examples, in bytes, as computed by the model handler's `get_num_bytes()` 
method. |
+| `inference_batch_latency_micro_secs` | Distribution | The time taken to 
perform inference on each batch of examples, measured in microseconds. |
+| `load_model_latency_milli_secs` | Distribution | The time taken to load and 
initialize the model, measured in milliseconds. |
+| `model_byte_size` | Distribution | The memory consumed to load and 
initialize the model, in bytes. RunInference measures it as the increase in the 
worker process's peak memory usage while the model loads. On platforms where 
the Python `resource` module isn't available, such as Windows, this metric is 
always 0. |
+{{< /table >}}
 
-* `inference_batch_latency_micro_secs_MEAN`: Represents the average time taken 
to perform inference across all batches of examples, measured in microseconds.
+Counters report a total, and distributions record one value at a time. For 
more information, see [Types of 
metrics](/documentation/programming-guide/#types-of-metrics). In the Google 
Cloud console, each distribution appears as four metrics with the suffixes 
`_COUNT`, `_MIN`, `_MAX`, and `_MEAN`, where `_COUNT` is the number of values 
recorded.

Review Comment:
   Just route to the metrics guide here, we don't need to cover the cloud 
specifics



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to