yixunzhang opened a new issue, #14947:
URL: https://github.com/apache/tvm/issues/14947
### Description
I am comparing the inference result of models with and without TensorRT as
the backend.
My simple PyTorch model `MatMul` was translated to operator nn.dense with an
input a relay.constant, and produces an all-zero result.
I referred to the test case "test_batch_matmul" in
tvm:tests/python/contrib/test_tensorrt.py. it works well with `matmul`, but
when I changed the operator into `dense` and a parameter to `constant`, it
produces the all-zero result.
PyTorch Model
```
class MatMul:
def __init__(self):
super(MatMul, self).__init__()
self.w = Parameter(data = torch.ones(1024, 1024))
def forward(self, x):
return torch.matmul(x, self.w)
```
### Expected behavior
Same result w/wo TensorRT。
### Actual behavior
The result produced by TensorRT is all-zero.
### Environment
TVM: 0.11.1
TensorRT: 8.2.0.6
Python 3.9.1
### Steps to reproduce
``` python
y_shape =(64,64)
data_type="float32"
target="cuda"
dev = tvm.device(target)
_y = tvm.nd.array(np.random.rand(*y_shape).astype(dtype=data_type))
y=relay.Constant(_y)
def get_graph(x_shape=(12, 128, 64), y_shape=(12,128,64), transa=True,
transb=True):
x = relay.var("x",shape=(x_shape), dtype="float32")
out = relay.nn.dense(x, y)
f=relay.Function([x],out)
return f, {"x":x_shape}, []
f, input_shapes, is_param = get_graph(x_shape, y_shape, transa, transb)
params = { x : np.random.uniform(-1,1,
input_shapes[x]).astype(dtype=data_type) for x in is_param }
input_dict ={
k: np.random.uniform(-1,1, v).astype(dtype=data_type)
for k, v in input_shapes.items()
if k not in is_param
}
result_dict = dict()
for mode in ["vm","graph"]:
for use_trt in [True, False]:
mod=tvm.IRModule()
mod["main"] = f
result_key = mode + (" trt" if use trt else "")
if use_trt:
use_fp16 = data_type == "float16"
trt_target = tvm.target.Target(f"tensorrt -use_fp16={use_fp16}")
mod = relay.transform.InferType()(mod)
mod = tensorrt.partition_for_tensorrt(mod,
params=params,target=trt_target)
print(mod)
with tvm.transform.PassContext(opt_level=3):
func = relay.create_executor(
mode, mod=mod,device=dev, target=[target, trt_target]
).evaluate()
else:
mod = relay.transform.InferType()(mod)
with tvm.transform.PassContext(opt_level=3):
func =relay.create_executor(
mode,mod=mod,device=dev,target=target
).evaluate()
result_dict[result_key] = func(**input_dict,**params)
print(f"{result_key}", result_dict[result_key].numpy() [:10,0])
```
### Triage
* byoc:tensorrt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
