denise-k opened a new issue #8610: URL: https://github.com/apache/tvm/issues/8610
# Context CI is taking a while (see #8552 and [build#1384](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/)), and I am tracking short-term fixes to alleviate this # Summary of the issue Each of the frontends in TVM is testing multiple common networks (e.g. `resnet_50, inception_v1, googlenet`). Functionally, the networks are the same once they get lowered into TVM, which means that there is a lot of duplicate coverage for each frontend. Each network test may only take seconds or minutes, but due to the ever-growing and overlapping `number of networks tested * number of frameworks supported`, this is now taking up over an hour of CI time. Here is a list of these network tests: | Frontend | Model | Runtime | | ---- | ---- | ----| | Tensorflow | [ssd](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.tensorflow/test_forward/Integration_Test___frontend__GPU___test_forward_ssd) | 11 min | | TensorFlow | [resnet v2](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.tensorflow/test_forward/Integration_Test___frontend__GPU___test_forward_resnetv2) | 2 min 30 sec | | TensorFlow | [ptb](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.tensorflow/test_forward/Integration_Test___frontend__GPU___test_forward_ptb) | 2 min 43 sec | | TensorFlow | [mobilenet](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.tensorflow/test_forward/Integration_Test___frontend__GPU___test_forward_mobilenet) | 43 sec | | PyTorch | [densenet 121](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_densenet121) | 2 min 32 sec | | PyTorch | [googlenet](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_googlenet) | 1 min 6 sec | | PyTorch | [inception v3](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_inception_v3) | 1 min 34 sec | | PyTorch | [mnasnet 0.5](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_mnasnet0_5) | 53 sec | | PyTorch | [mobilenet v2](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_mobilenet_v2) | 49 sec | | PyTorch | [resnet18](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_resnet18) | 30 sec | | PyTorch | [segmentation](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_segmentation_models) | 3 min 52 sec | | PyTorch | [squeezenet 1.0](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_squeezenet1_0) | 17 sec | | PyTorch | [squeezenet 1.1](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.pytorch/test_forward/Integration_Test___frontend__GPU___test_squeezenet1_1) | 13 sec | | Caffe | [inception v1](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.caffe/test_forward/Integration_Test___frontend__CPU___test_forward_Inceptionv1) | 31 sec | | Caffe | [mobilenet v2](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.caffe/test_forward/Integration_Test___frontend__CPU___test_forward_Mobilenetv2) | 31 sec | | Caffe | [resnet 50](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.caffe/test_forward/Integration_Test___frontend__CPU___test_forward_Resnet50) | 44 sec | | ONNX | [densenet](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.onnx/test_forward/Integration_Test___frontend__GPU___test_densenet) | 2 min 2 sec | | CoreML | [mobilenet](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.coreml/test_forward/Integration_Test___frontend__GPU___test_mobilenet_checkonly) | 33 sec | | CoreML | [resnet](https://ci.tlcpack.ai/job/tvm/job/main/1384/testReport/cython.tests.python.frontend.coreml/test_forward/Integration_Test___frontend__GPU___test_resnet50_checkonly) | 1 min 12 sec | # Suggested actions * Separate out full network test cases from frontend tests * Make sure critical networks are tested in integration tests * Move full-network frontend tests to a nightly or non-PR-gating CI pipeline -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
