cbalint13 commented on PR #19851: URL: https://github.com/apache/tvm/pull/19851#issuecomment-4761293756
> in this case, i think we should deliberately keep the function in disco because remote may need to load module this way. for local c++ applications, once can simply use the more detailed sequence of vm_load_executable/vm_initialization. The DSO cache is also very specific to disco i think to make sure no reload but may not be general applicable @tqchen , Permit the following argues: * Disco will still fully have it [here](https://github.com/apache/tvm/pull/19851/changes#diff-4fe524fb65fb3f8be1c128a138cf9f54b9311fbe0d0ec0c9a02ed677c61576a0R93-R98), and the cache can be also moved here, so can remain a disco exclusive feature. * At this moment, disco is invoked only from python via that mentioned ffi interface, I found no other references. --- Currently there is no way (at least i found none) to do c++ inference except this bit awkward way: ``` #include <runtime/vm/vm.h> #include <ffi/extra/module.h> #include <runtime/vm/executable.h> #include <runtime/device_api.h> int main() { tvm::Device dev = {kDLCPU, 0}; auto mod_dso = tvm::ffi::Module::LoadFromFile("./lib.tar.so"); auto vme = mod_dso->GetFunction("vm_load_executable", false); auto mod = (*vme)().cast<tvm::ffi::Module>(); tvm::ffi::Optional<tvm::ffi::Function> vm_initialization = mod->GetFunction("vm_initialization"); (*vm_initialization)(static_cast<int>(dev.device_type), static_cast<int>(dev.device_id), static_cast<int>(tvm::runtime::AllocatorType::kPooled), static_cast<int>(dev.device_type), 0, static_cast<int>(tvm::runtime::AllocatorType::kPooled)); auto input_tensor = tvm::runtime::Tensor::Empty({1, 576}, tvm::runtime::DataType::Float(32), dev); auto sr_input = tvm::runtime::Tensor::Empty({}, tvm::runtime::DataType::Int(64), dev); static_cast<int64_t*>(sr_input->data)[0] = 16000; auto state = tvm::runtime::Tensor::Empty({2, 1, 128}, tvm::runtime::DataType::Float(32), dev); tvm::ffi::Optional<tvm::ffi::Function> vm_set_input = mod->GetFunction("set_input"); (*vm_set_input)("main", input_tensor, sr_input, state); tvm::ffi::Optional<tvm::ffi::Function> vm_invoke_stateful = mod->GetFunction("invoke_stateful"); (*vm_invoke_stateful)("main"); tvm::ffi::Optional<tvm::ffi::Function> vm_get_output = mod->GetFunction("get_output"); tvm::ffi::Any dprbs = (*vm_get_output)("main", 0); tvm::ffi::Any dstates = (*vm_get_output)("main", 1); return 0; } ``` To make it simpler we could rapidly obtain the module this way, which is a leap forward: ``` - #include <runtime/vm/executable.h> - - auto mod_dso = tvm::ffi::Module::LoadFromFile("./lib.tar.so"); - auto vme = mod_dso->GetFunction("vm_load_executable", false); - - auto mod = (*vme)().cast<tvm::ffi::Module>(); - tvm::ffi::Optional<tvm::ffi::Function> vm_initialization = mod->GetFunction("vm_initialization"); - (*vm_initialization)(static_cast<int>(dev.device_type), static_cast<int>(dev.device_id), - static_cast<int>(tvm::runtime::AllocatorType::kPooled), - static_cast<int>(dev.device_type), 0, - static_cast<int>(tvm::runtime::AllocatorType::kPooled)); + auto mod = tvm::runtime::vm::LoadVMModule("./lib.tar.so", dev); ``` , and then follow the rest (will see after how could simplify the second part too). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
