tvalentyn commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1137906112
##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
#
test_1:
- test_name:
apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 152
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet152
project: apache-beam-testing
metric_name: mean_load_model_latency_milli_secs
labels:
- run-inference
- # Optional parameters.
- min_runs_between_change_points: 3
- num_runs_in_change_point_window: 30
test_2:
- test_name:
apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 152
+ metrics_dataset: beam_run_inference
+ metrics_table: torch_inference_imagenet_results_resnet152
+ project: apache-beam-testing
+ metric_name: mean_inference_batch_latency_micro_secs
+ labels:
+ - run-inference
+
+test_3:
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 101
+ metrics_dataset: beam_run_inference
+ metrics_table: torch_inference_imagenet_results_resnet101
+ project: apache-beam-testing
+ metric_name: mean_load_model_latency_milli_secs
+ labels:
+ - run-inference
+
+test_4:
+ test_name: Pytorch image classification on 50k images of size 224 x 224 with
resnet 101
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet101
project: apache-beam-testing
+ metric_name: mean_inference_batch_latency_micro_secs
+ labels:
+ - run-inference
+
+test_5:
+ test_name: TFT Criteo 10 GB no shuffle test
+ metrics_dataset: beam_cloudml
+ metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+ project: apache-beam-testing
+ metric_name: runtime_sec
+ labels:
+ - python_tft_criteo
+
+test_6:
+ test_name: TFT Criteo 10 GB test
+ metrics_dataset: beam_cloudml
+ metrics_table: cloudml_benchmark_criteo_10GB
+ project: apache-beam-testing
+ metric_name: runtime_sec
+ labels:
+ - python_tft_criteo
+
+test_7:
+ test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on
n1-standard-4 machine
Review Comment:
50 is too many workers for this test, IMO. also, # of workers not indicated
in other test cases, so maybe remove it here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
