areusch opened a new issue #8696: URL: https://github.com/apache/tvm/issues/8696
Right now when AutoTVM and auto_scheduler builders and runners encounter exception conditions, the exceptions are caught and stored into result objects, but never printed. This is WAI when the log level is e.g. below logging.INFO, because during autotuning, some schedules are expected to fail to build and autotuning should be robust to that. However, there are a lot of valid debugging scenarios (e.g. debugging new schedules, debugging targets, debugging changes to autotvm, debugging failed tests) where currently, you first need to modify `python/tvm/autotvm` or `python/tvm/auto_scheduler` code to print out the exception. The task here is twofold, the second part could be deferred: part 1: - [ ] add logging.debug() calls to autotvm and auto_scheduler to print the exception - [ ] add unit tests to verify these get called call sites to log exceptions from in autotvm: 1. [builder](https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/local_executor.py#L55) 2. [runner](https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/measure_methods.py#L618) 3. [establishing the session](https://github.com/apache/tvm/blob/main/python/tvm/autotvm/measure/measure_methods.py#L583) (kind of the same as the previous item, but a different class of errors can occur here--those errors raised from `session_constructor`) part 2: do this more scalably for autotvm, and provide a way to inspect the autotvm results: - [ ] improve the return value of autotvm: output some kind of class that summarizes e.g. # successful trials, # failed trials, tune log results by trial, exceptions by trial - [ ] determine the appropriate way to serialize this to disk for consumption from `tvmc` - [ ] add testing for the above @mikepapadim if you have time for at least part 1, that would be quite helpful! cc @jroesch @tqchen @leandron @denise-k -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
