AnandInguva commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1138788032
##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
#
test_1:
- test_name:
apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 152
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet152
project: apache-beam-testing
metric_name: mean_load_model_latency_milli_secs
labels:
- run-inference
- # Optional parameters.
- min_runs_between_change_points: 3
- num_runs_in_change_point_window: 30
test_2:
- test_name:
apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 152
+ metrics_dataset: beam_run_inference
+ metrics_table: torch_inference_imagenet_results_resnet152
+ project: apache-beam-testing
+ metric_name: mean_inference_batch_latency_micro_secs
+ labels:
+ - run-inference
+
+test_3:
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 101
+ metrics_dataset: beam_run_inference
+ metrics_table: torch_inference_imagenet_results_resnet101
+ project: apache-beam-testing
+ metric_name: mean_load_model_latency_milli_secs
+ labels:
+ - run-inference
+
+test_4:
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 101
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet101
project: apache-beam-testing
+ metric_name: mean_inference_batch_latency_micro_secs
+ labels:
+ - run-inference
+
+test_5:
+ test_name: TFT Criteo 10 GB no shuffle test
+ metrics_dataset: beam_cloudml
+ metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+ project: apache-beam-testing
+ metric_name: runtime_sec
+ labels:
+ - python_tft_criteo
+
+test_6:
+ test_name: TFT Criteo 10 GB test
+ metrics_dataset: beam_cloudml
+ metrics_table: cloudml_benchmark_criteo_10GB
+ project: apache-beam-testing
+ metric_name: runtime_sec
+ labels:
+ - python_tft_criteo
+
+test_7:
+ test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on
n1-standard-4 machine
Review Comment:
This was the same configurations used in the tft criteo tests in google3.
mentioning 50 workers indicates we use 50 machines rather than autoscaling.
if 50 is too many for this test, we can change that in a different PR and
disable the config here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
