echuraev commented on pull request #8492: URL: https://github.com/apache/tvm/pull/8492#issuecomment-912558170
@tqchen, @vinx13 I tried to do that (passing callable function as an argument) and specially I commited these changes on my local branch: https://github.com/echuraev/tvm/tree/echuraev/fix_measure_build_func But it doesn't work. I got the following error: ``` Get 64 programs to measure: Traceback (most recent call last): File "exp_nn/tune_my_model.py", line 148, in <module> runner() File "exp_nn/tune_my_model.py", line 143, in runner run_tuning(tasks, task_weights, log_file) File "exp_nn/tune_my_model.py", line 93, in run_tuning tuner.tune(tune_option) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/auto_scheduler/task_scheduler.py", line 357, in tune self._tune_task(idx) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/auto_scheduler/task_scheduler.py", line 452, in _tune_task measure_inputs, measure_results = self.search_policies[task_idx].continue_search_one_round( File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/auto_scheduler/search_policy.py", line 119, in continue_search_one_round return _ffi_api.SearchPolicyContinueSearchOneRound(self, num_measure, measurer) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__ raise get_last_ffi_error() ValueError: Traceback (most recent call last): [bt] (8) 9 libtvm.dylib 0x000000011d4eb681 tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<tvm::runtime::Array<tvm::auto_scheduler::MeasureInput, void> const&, int&, int&, tvm::runtime::PackedFunc&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, int&>(tvm::runtime::Array<tvm::auto_scheduler:: MeasureInput, void> const&, int&, int&, tvm::runtime::PackedFunc&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, int&) const + 401 [bt] (7) 8 libtvm.dylib 0x000000011d447b71 std::__1::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 65 [bt] (6) 7 libtvm.dylib 0x000000011d447d9f std::__1::__function::__value_func<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) const + 95 [bt] (5) 6 libtvm.dylib 0x000000011f707d86 std::__1::__function::__func<TVMFuncCreateFromCFunc::$_2, std::__1::allocator<TVMFuncCreateFromCFunc::$_2>, void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 70 [bt] (4) 5 libtvm.dylib 0x000000011f7092c7 std::__1::__function::__alloc_func<TVMFuncCreateFromCFunc::$_2, std::__1::allocator<TVMFuncCreateFromCFunc::$_2>, void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 71 [bt] (3) 4 libtvm.dylib 0x000000011f709317 void std::__1::__invoke_void_return_wrapper<void, true>::__call<TVMFuncCreateFromCFunc::$_2&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(TVMFuncCreateFromCFunc::$_2&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 71 [bt] (2) 3 libtvm.dylib 0x000000011f7093b3 decltype(std::__1::forward<TVMFuncCreateFromCFunc::$_2&>(fp)(std::__1::forward<tvm::runtime::TVMArgs>(fp0), std::__1::forward<tvm::runtime::TVMRetValue*>(fp0))) std::__1::__invoke<TVMFuncCreateFromCFunc::$_2&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(TVMFuncCreateFromCFunc::$_2&, tvm::runtime::T VMArgs&&, tvm::runtime::TVMRetValue*&&) + 115 [bt] (1) 2 libtvm.dylib 0x000000011f70947f TVMFuncCreateFromCFunc::$_2::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 175 [bt] (0) 1 libtvm.dylib 0x000000011f74cb55 tvm::runtime::Backtrace() + 37 File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 81, in cfun rv = local_pyfunc(*pyargs) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/auto_scheduler/measure.py", line 697, in local_builder_build for res in tuple_res: File "/Users/echuraev/opt/anaconda3/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator yield fs.pop().result() File "/Users/echuraev/opt/anaconda3/lib/python3.8/concurrent/futures/_base.py", line 439, in result return self.__get_result() File "/Users/echuraev/opt/anaconda3/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception File "/Users/echuraev/opt/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/contrib/popen_pool.py", line 390, in <lambda> worker = lambda x: self._worker_run_with_error_catching(fn, (x,), None) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/contrib/popen_pool.py", line 343, in _worker_run_with_error_catching return MapResult(status=StatusKind.COMPLETE, value=self._worker_run(fn, args, kwargs)) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/contrib/popen_pool.py", line 338, in _worker_run proc.send(fn, args, kwargs, self._timeout) File "/Users/echuraev/Workspace/OctoML/tvm/python/tvm/contrib/popen_pool.py", line 224, in send data = cloudpickle.dumps((fn, args, kwargs, timeout), protocol=pickle.HIGHEST_PROTOCOL) File "/Users/echuraev/opt/anaconda3/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps cp.dump(obj) File "/Users/echuraev/opt/anaconda3/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump return Pickler.dump(self, obj) ValueError: ctypes objects containing pointers cannot be pickled ``` One interesting thing that I tried to dump each of these parameters in tuple `(fn, args, kwargs, timeout)` separately and I didn't see this error. I suppose that the problem is happened because `fn` is a `PackedFunction` with python function. And CloudPickle is not able to serialize native `PackedFunction` object. Another idea how we can quick fix this problem: move `tar` and `ndk` functions to the `_local_build_worker` and pass only names "tar" or "ndk" to it and select implementation there. In this case it would work for current tvm defined custom functions. In case when passed name is not "tar" or "ndk", then we will have another argument with callable function. It won't work now. Because of this problem with packed functions but we can fix it in the next PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
