anwang2009 opened a new issue #4304: [RFC] Benchmark Performance Log Format WIP URL: https://github.com/apache/incubator-tvm/issues/4304 # Workload record storage proposal This is a migration and summation of the discussion https://discuss.tvm.ai/t/rfc-benchmark-performance-log-format/4610/15. Please head over there to put in your two cents. This is a WIP issue maintained for visibility. We propose a column-based data storage format that unifies how TVM results are tracked. For now the only metric we suggest tracking is execution time, with a focus on reproducibility. We foresees the need for a standard record format to pinpoint the most crucial target areas for future development in TVM. ## Use cases and motivation - Easier cross-reference to other frameworks/runs - Experiment reproduction - Building nightly perf regression - Standardization will separate logging and visualization efforts ## Key points to consider - Maintain reproducibility of experiments based solely on an output record - Granularity of metrics -- e.g. per-layer vs end-to-end - config/misc json objects -- enforce prescriptive keys or keep it flexible for now? - Extensibility of this schema towards desired features in the future ## Proposed format |header|examples|category|notes/justification| |---|---|---|---| |timestamp|1572282699.6|metadata|Indicates when this record is logged| |schema_version|0.1|metadata|Ensure reproducibility as we iterate on this schema| |additional_info|{ build_start:, host_env:, ... }|metadata|`docker_tag`, `docker_hash` are required; `device_memory_usage`, `code_size`, `validation_accuracy`, `compile_time` are optionally specified| |workload|ai.onnx.resnet-18|workload|(1) |workload_args|{{“input_shape”: (3, 224, 224)}, {“args_shape”: [list_of_shape]}, {“data_layout”: NHCW}}|workload| |engine|tvm / onnx-runtime|compiler| |engine_version|0.5 / 996cf30e8d54b4bf58f0c9950475f47bba7e2c7e|compiler|Include either version or SHA, or both in the format "0.5:996cf3..." |engine_config|{“llvm”: “llvm-8”, “nvcc”: 10.1, "accelerator": "MLAS"}|compiler|fields are optionally specified| |target|llvm -mcpu=skylake-avx512|compiler|compilation target, used for single platform| |build_config|{"opt_level": 3, "layer_schedules": [...]| |env_name|aws.a1.large / rapi3b / iphone11pro|hardware|Readable cloud vendor category or hardware, mobile phone name| |hw_model|Intel Xeon E5-2686 v4 / bcm2837/ apple-a13|hardware|Hardware type, soc type| |runtime_config|{“num_cpu_threads”: 3, “cudnn”: “cudnn-8”, “cuda_driver”: “480.10.1”, “os”: linux}|hardware|Backend runtime arguments| |execution_config|{“number”: 1, “repeat”: 10, “min_repeat_ms”, 0}|statistics|workload execution parameters| |execution_elapsed_ms_mean|float32|statistics| |execution_elapsed_ms_std|float32|statistics| 1. Namespacing: Use strings of the form “domain.name” where domain and name are any valid string except for the domains of ‘topi’ and ‘relay’ which are reserved to only refer to TOPI and Relay operator names respectively.\ For topi and relay namespaces, the operator name may be followed by an optional “:\<SHA\>” which corresponds to a particular git SHA of the TVM repository. ## Possible future schema directions - Add more metrics to execution statistics, with statistics collection components enabled by some config object
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
