cbalint13 commented on PR #19863: URL: https://github.com/apache/tvm/pull/19863#issuecomment-4767634950
> On reference evaluator, my guess is reference evaluator would even be slower because it generates reference results and also is integration runs, so it would be worse than onxxruntime in terms of ci time. > > I think the main issue is to consider the balance of ci pressure and our core features. Ideally the invariance are: > > * For importers, we cross validate the ingestion correctness via stuctural check > * We have some levels of correctness checks in op UT, which might help > > An altnertative could be we split out the frontend modules into separate repos, but still there would be an ci pressure concern there Despite its pure python its pretty lightweight, not slower, I was also surprised. I give you a ruff number: ~514 tests (include ops all own versions) runs in ~20sec on the [example](https://github.com/cbalint13/onnx2mlir/blob/6df5c9d36fa5de6acf883fdc10df5bcc910e035f/tests/python/conversion/test_onnx_to_linalg_ops.py#L289-L345) , in this 20 sec the lowering to mlir and instantiating the mlir runtime to compute the operator is also included. I am up for a task to see if we can benefit "ReferenceEvaluator" over ```onnxruntime```, can migrate all testcases within a Draft PR to see numbers, I dont mind if experiment proves the otherwise and we drop the idea. Let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
