[GitHub] [tvm] AndrewZhaoLuo commented on pull request #13654: [ONNX] Add converter for QAttention from Microsoft onnxruntime contrib opset
AndrewZhaoLuo commented on PR #13654: URL: https://github.com/apache/tvm/pull/13654#issuecomment-1366442483 Apologies, I've been quite sick. I'll try to look at this Thursday. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] elvin-n commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
elvin-n commented on code in PR #13621:
URL: https://github.com/apache/tvm/pull/13621#discussion_r1058100234
##
python/tvm/relay/op/strategy/x86.py:
##
@@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type,
target):
if (
not attrs.transpose_a
and attrs.transpose_b
-and target_has_vnni(mcpu)
+and target_has_avx512(mcpu)
and inputs[0].dtype == "uint8"
and inputs[1].dtype == "int8"
and inputs[1].shape[-2] % 16 == 0
and inputs[1].shape[-1] % 4 == 0
):
strategy.add_implementation(
-wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute,
need_out_dtype=True),
-wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni),
-name="batch_matmul_vnni.x86",
Review Comment:
I would consider amx vs vnni avx512 avx2 sse3 (btw, there is no sse2 for
int8, required instructions appeared if I am not mistaken in sse3.x) because
first is matrix multiplication, other ones are vector instructions. For now I
propose to go from local to generic and when we see needs in differentiate
vector sets, we will do this. For now pattern look similar for all of vector
instructions, the aspect of blocking should be added separately if it is not
done yet, The aspect of lanes in TVM intrinsic should be covered in this PR
> match inner loops to these varying sizes.
The inner loop is the same for all these instructions. It will be
```
for (int k = 0; k < 4; k++){
output[i] += data[k] * kernel[i][k]
}
```
> TVM is a compiler after all, to my knowledge the only capable of
auto-tensorization with arbitrary intrinsic.
again, I propose to move from local to generic patterns. We do not limit
anything for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tvm] echuraev commented on a diff in pull request #13654: [ONNX] Add converter for QAttention from Microsoft onnxruntime contrib opset
echuraev commented on code in PR #13654:
URL: https://github.com/apache/tvm/pull/13654#discussion_r1058087546
##
python/tvm/relay/frontend/onnx.py:
##
@@ -1379,6 +1379,298 @@ def massage(tensor):
return _expr.TupleWrapper(_expr.Tuple([output, present]), 2)
+class QAttention(OnnxOpConverter):
+"""Operator converter for QAttention from Microsoft onnxruntime contrib
opset.
+
+This is the self-attention mechanism used in transformer models.
+"""
+
+@classmethod
+def _impl_v1(cls, inputs, attr, params):
+# * Read attrs *
+num_heads = attr["num_heads"]
+unidirectional = attr["unidirectional"]
+
+# * Read inputs *
+# (batch, seq, in_hidden)
+input_emb = inputs[0]
+
+# (in_hidden, 3 * out_hidden), where out_hidden = num_heads * head_size
+weight = inputs[1]
+
+# (3 * out_hidden,)
+bias = inputs[2]
+
+# Scalar, which means a per-tensor/layer quantization
+input_scale = inputs[3]
+
+# Scalar or a 1D tensor, which means a per-tensor/per-column
quantization.
+# Its size should be 3 * out_hidden if it is per-column quantization
+weight_scale = inputs[4]
+
+# TODO(agladyshev):
+# ORT documentation says that shape is (batch,),
+# but in ORT source code we have following comment:
+# 1. (batch_size)
+# 2. (2 * batch_size)
+# 3. (batch_size, 1)
+# 4. (1, 1)
+# 5. (batch_size, past_sequence_length + sequence_length)
+# In practice, for GPT-2 there shape is (batch, past_seq_length +
seq_length).
+# Currently only (batch, past_seq_length + seq_length) shape is
supported.
+mask_index = inputs[5]
+
+# Scalar, which means a per-tensor/layer quantization
Review Comment:
nit: You have absolutely the same comment for `input[3]` and `input[7]`
##
python/tvm/relay/frontend/onnx.py:
##
@@ -1379,6 +1379,298 @@ def massage(tensor):
return _expr.TupleWrapper(_expr.Tuple([output, present]), 2)
+class QAttention(OnnxOpConverter):
+"""Operator converter for QAttention from Microsoft onnxruntime contrib
opset.
+
+This is the self-attention mechanism used in transformer models.
+"""
+
+@classmethod
+def _impl_v1(cls, inputs, attr, params):
+# * Read attrs *
+num_heads = attr["num_heads"]
+unidirectional = attr["unidirectional"]
+
+# * Read inputs *
+# (batch, seq, in_hidden)
+input_emb = inputs[0]
+
+# (in_hidden, 3 * out_hidden), where out_hidden = num_heads * head_size
+weight = inputs[1]
+
+# (3 * out_hidden,)
+bias = inputs[2]
+
+# Scalar, which means a per-tensor/layer quantization
+input_scale = inputs[3]
+
+# Scalar or a 1D tensor, which means a per-tensor/per-column
quantization.
+# Its size should be 3 * out_hidden if it is per-column quantization
+weight_scale = inputs[4]
+
+# TODO(agladyshev):
+# ORT documentation says that shape is (batch,),
+# but in ORT source code we have following comment:
+# 1. (batch_size)
+# 2. (2 * batch_size)
+# 3. (batch_size, 1)
+# 4. (1, 1)
+# 5. (batch_size, past_sequence_length + sequence_length)
+# In practice, for GPT-2 there shape is (batch, past_seq_length +
seq_length).
+# Currently only (batch, past_seq_length + seq_length) shape is
supported.
+mask_index = inputs[5]
+
+# Scalar, which means a per-tensor/layer quantization
+input_zero_point = inputs[6]
+
+# Scalar or a 1D tensor, which means a per-tensor/per-column
quantization.
+# Its size should be 3 * out_hidden if it is per-column quantization
+weight_zero_point = inputs[7]
+
+# (2, batch, num_heads, past_seq, head_size)
+past = inputs[8]
+
+# * Parse inputs *
+t1 = ["int8", "uint8"]
+t2 = ["int8", "uint8"]
+t3 = ["float32", "float16"]
+t4 = ["int32"]
+
+# input
+assert infer_type(input_emb).checked_type.dtype in t1
+assert (
+len(infer_shape(input_emb)) == 3
+), "Input should be 3D tensor with shape (batch_size, sequence_length,
input_hidden_size)"
+(batch_size, seq_len, input_hidden) = infer_shape(input_emb)
+assert input_hidden > 0, (
+"The weight tensor has (input_hidden_size, 3 * output_hidden_size)
shape, so it doesn't"
+f" make sense to have ({input_hidden}, 3 * output_hidden_size)
weight tensor."
+)
+
[GitHub] [tvm] peiwenYe opened a new issue, #13667: Code does not match comments
peiwenYe opened a new issue, #13667: URL: https://github.com/apache/tvm/issues/13667 In python/tvm/topi/nn/conv3d_transpose.py, code does not match the comments  If all the output kernel layout should be OIDHW? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[tvm] branch main updated: [CLML][RELAY] Enable Pad and Conv2d layer fusion (#13649)
This is an automated email from the ASF dual-hosted git repository.
srk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new ece99a243b [CLML][RELAY] Enable Pad and Conv2d layer fusion (#13649)
ece99a243b is described below
commit ece99a243beab1fe879d78868367731d5a516a83
Author: krishnaraj36 <45380557+krishnara...@users.noreply.github.com>
AuthorDate: Wed Dec 28 11:24:11 2022 +0530
[CLML][RELAY] Enable Pad and Conv2d layer fusion (#13649)
* [CLML][RELAY] Enable Pad and Conv2d layer fusion
Enabled clml supported nn.pad+nn.conv2d fusion pattern in clml pattern table
* Fix pad testcase attributes
* Fix the lint error
* Fix the lint error
* Removed redundent check in clml pattern
* Fix the lint error
Co-authored-by: kvegiraj
---
python/tvm/relay/op/contrib/clml.py| 21 +
src/relay/backend/contrib/clml/codegen.cc | 2 +-
tests/python/contrib/test_clml/test_ops.py | 4 ++--
3 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/python/tvm/relay/op/contrib/clml.py
b/python/tvm/relay/op/contrib/clml.py
index c3d4eb8470..6453b8a06c 100644
--- a/python/tvm/relay/op/contrib/clml.py
+++ b/python/tvm/relay/op/contrib/clml.py
@@ -147,6 +147,23 @@ def clml_pattern_table():
pattern = pattern.optional(is_op("clip"))
return pattern
+def pad_conv_pattern():
+"""Create a pad with convolution pattern."""
+pattern = is_op("nn.pad")(wildcard(), is_constant())
+pattern = is_op("nn.conv2d")(pattern, is_constant())
+pattern = pattern.optional(lambda x: is_op("nn.bias_add")(x,
is_constant()))
+pattern = pattern.optional(lambda x: is_op("add")(x, is_constant()))
+pattern = pattern.optional(
+lambda x: is_tuple_get_item(
+is_op("nn.batch_norm")(
+x, is_constant(), is_constant(), is_constant(),
is_constant()
+)
+)
+)
+pattern = pattern.optional(is_op("nn.relu"))
+pattern = pattern.optional(is_op("clip"))
+return pattern
+
def batch_norm_pattern():
"""Create a batch norm pattern."""
pattern = is_op("nn.batch_norm")(
@@ -200,9 +217,11 @@ def clml_pattern_table():
while call.op.name != "nn.conv2d":
call = call.args[0]
+
attrs, args = call.attrs, call.args
if attrs.data_layout != "NCHW":
return False
+
if (
(not clip_found)
and (attrs.kernel_size[0] == 3)
@@ -211,6 +230,7 @@ def clml_pattern_table():
and (attrs.channels == attrs.groups)
):
return False
+
data_typ = args[0].checked_type
kernel_typ = args[1].checked_type
is_depthwise = is_depthwise_conv2d(
@@ -246,6 +266,7 @@ def clml_pattern_table():
return True
return [
+("clml.pad_conv2d", pad_conv_pattern(), check_conv),
("clml.conv2d", conv_pattern(), check_conv),
("clml.dense", dense_pattern(), check_default_op),
("clml.pad", pad_pattern(), check_pad_op),
diff --git a/src/relay/backend/contrib/clml/codegen.cc
b/src/relay/backend/contrib/clml/codegen.cc
index 9ecec0c453..167c48e1ba 100644
--- a/src/relay/backend/contrib/clml/codegen.cc
+++ b/src/relay/backend/contrib/clml/codegen.cc
@@ -83,7 +83,7 @@ class CLMLJSONSerializer : public
backend::contrib::JSONSerializer {
ICHECK(comp.defined()) << "CLML JSON runtime only supports composite
functions.";
const std::string name = comp.value();
std::shared_ptr json_node;
-if (name == "clml.conv2d") {
+if (name == "clml.conv2d" || name == "clml.pad_conv2d") {
json_node = CreateCompositeConvJSONNode(cn);
} else if (name == "clml.batch_norm") {
json_node = CreateBatchNormJSONNode(cn);
diff --git a/tests/python/contrib/test_clml/test_ops.py
b/tests/python/contrib/test_clml/test_ops.py
index d2431d2dfd..da09715fbe 100644
--- a/tests/python/contrib/test_clml/test_ops.py
+++ b/tests/python/contrib/test_clml/test_ops.py
@@ -45,7 +45,7 @@ def _get_conv_model(
a = relay.var(next(iter(var)), shape=shape, dtype=dtype)
input_arr = var[next(iter(var))]
if has_pad:
-p = ((0, 0), (padding[0], padding[0]), (padding[1], padding[1]), (0,
0))
+p = ((0, 0), (0, 0), (padding[0], padding[0]), (padding[1],
padding[1]))
a = relay.nn.pad(a, pad_width=p)
padding = (0, 0, 0, 0)
else:
@@ -97,7 +97,7 @@ def test_conv2d(device, dtype):
trials = [
# Normal convolution
[3, 3, (1, 1), (1, 1), (1, 1), 4, (14, 10, 10), (False, False, False)],
-[2, 1, (2, 2), (1, 1), (1, 1), 7, (15, 16, 12), (False, False, True)],
+[2, 1, (2, 2), (1, 1), (1, 1), 7, (15, 16, 12), (True, False, True)],
[3, 3, (2, 1),
[GitHub] [tvm] srkreddy1238 merged pull request #13649: [CLML][RELAY] Enable Pad and Conv2d layer fusion
srkreddy1238 merged PR #13649: URL: https://github.com/apache/tvm/pull/13649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] echuraev commented on a diff in pull request #13659: [MetaSchedule] Add "disabled_pass" option in tuning API
echuraev commented on code in PR #13659: URL: https://github.com/apache/tvm/pull/13659#discussion_r1058071985 ## python/tvm/meta_schedule/relay_integration.py: ## @@ -250,6 +252,7 @@ def tune_relay( seed: Optional[int] = None, module_equality: str = "structural", num_tuning_cores: Union[Literal["physical", "logical"], int] = "physical", +disabled_pass: Optional[Union[List[str], Set[str], Tuple[str]]] = None, Review Comment: Could you please add information about this parameter to the docstring? ## python/tvm/meta_schedule/relay_integration.py: ## @@ -345,6 +350,7 @@ def compile_relay( } ), executor: Optional["relay.backend.Executor"] = None, +disabled_pass: Optional[Union[List[str], Set[str], Tuple[str]]] = None, Review Comment: Add information to doc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[tvm] branch nightly updated (e2680142ef -> f83055f90a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly in repository https://gitbox.apache.org/repos/asf/tvm.git from e2680142ef [CMSIS-NN] Global function that provides range based on dtype (#13652) add 7a38477b2f [Pytorch][Relay] aten::_weight_norm implementation (#13661) add 520f2c594b [Relay][Frontend] Span filling common API (#13402) add f83055f90a [QNN] Change in Pass Context for lookup table calculation (#13660) No new revisions were added by this update. Summary of changes: python/tvm/relay/expr.py | 202 +++--- python/tvm/relay/frontend/common.py | 165 - python/tvm/relay/frontend/pytorch.py | 15 ++ python/tvm/relay/function.py | 7 +- python/tvm/relay/loops.py | 2 +- python/tvm/relay/qnn/op/canonicalizations.py | 23 ++- python/tvm/testing/utils.py | 22 +++ src/ir/span.cc| 4 + src/relay/ir/expr.cc | 88 +-- src/relay/ir/function.cc | 4 +- tests/python/frontend/pytorch/test_forward.py | 24 +++ tests/python/frontend/test_common.py | 194 - tests/python/relay/utils/tag_span.py | 108 ++ 13 files changed, 808 insertions(+), 50 deletions(-) create mode 100644 tests/python/relay/utils/tag_span.py
[GitHub] [tvm] wangzy0327 opened a new issue, #13666: [Bug] rocm platform result are not correct
wangzy0327 opened a new issue, #13666:
URL: https://github.com/apache/tvm/issues/13666
I tried to execute mnist-model by tvm in rocm platform(rocm 5.2). The result
of execution is error.
### Expected behavior
The result of rocm platform equals result of cuda platform or opencl platform
### Actual behavior
The result of rocm platform not equal result of cuda platform or opencl
platform
### Environment
Operating System:Ubuntu 20.04
TVM version : 7f1856d34f03113dc3a7733c010be43446161944
platform: rocm 5.2
Any environment details, such as: Operating System, TVM version, etc
### Steps to reproduce
There is the test code.
onnx_rocm.py
```
from csv import writer
from pyexpat import model
import onnx
#from tvm.driver import tvmc
import numpy as np
import tvm
import tvm.relay as relay
from tvm.contrib import graph_executor
import tvm.testing
import numpy as np
import os
class NetworkData():
def
__init__(self,name:str,net_set:list,prefix_str:str,suffix_str:str,input_name:str,input:tuple,output:tuple):
self.name = name
self.net_set = net_set
self.prefix_str = prefix_str
self.suffix_str = suffix_str
self.input_name = input_name
self.input = input
self.output = output
mnist_networkData = NetworkData(name = "mnist",
net_set = ["mnist-7","mnist-8"],
prefix_str = "mnist/model/",
suffix_str = ".onnx",
input_name = 'Input3',
input = (1,1,28,28),
output = (1,10))
MODEL_NAME = {
"mnist":mnist_networkData,
}
dtype="float32"
common_prefix_str = "onnx-model/vision/classification/"
tol_paras = [1e-7,1e-6,1e-5,1e-4,1e-3,1e-2]
import logging
logging.basicConfig(level=logging.ERROR)
import warnings
warnings.filterwarnings('ignore')
def build(target:str,mod:tvm.IRModule, params:dict, input_name:str,
input_data:np.ndarray, input:tuple, output: tuple) -> np.ndarray:
tgt = tvm.target.Target(target=target, host="llvm")
with tvm.transform.PassContext(opt_level=3):
lib = relay.build(mod, target=target, params=params)
# print(lib.get_lib().imported_modules[0].get_source())
# print("source code
start")
# print(lib.get_lib().imported_modules[0].get_source())
# print("source code
end")
dev = tvm.device(str(target), 0)
module = graph_executor.GraphModule(lib["default"](dev))
module.set_input(input_name, input_data)
module.run()
output_shape = output
tvm_output = module.get_output(0, tvm.nd.empty(output_shape)).numpy()
return tvm_output
def main(model_network : NetworkData):
# 设置随机种子
np.random.seed(0)
I_np = np.random.uniform(size = model_network.input).astype(dtype)
print(I_np[0][0][0][:10])
header =
['network_name','network_sub_name','input','output','tolerance','rocm_cost_time','opencl_cost_time']
rows = []
for child_model_network in model_network.net_set:
print(""+child_model_network+"start-")
onnx_model = onnx.load(common_prefix_str +
model_network.prefix_str +
child_model_network +
model_network.suffix_str)
shape_dict = {model_network.input_name: I_np.shape}
mod, params = relay.frontend.from_onnx(onnx_model, shape_dict)
import datetime
# opencl_starttime = datetime.datetime.now()
# opencl_output = build("opencl",mod = mod,params =
params,input_name = model_network.input_name,input_data = I_np, input =
I_np.shape, output = model_network.output)
# opencl_endtime = datetime.datetime.now()
# opencl_duringtime = opencl_endtime - opencl_starttime
# print("%15s network opencl cost time is %s
s"%(child_model_network,opencl_duringtime))
rocm_starttime = datetime.datetime.now()
rocm_output = build("rocm",mod = mod,params = params,input_name =
model_network.input_name,input_data = I_np, input = I_np.shape, output =
model_network.output)
rocm_endtime = datetime.datetime.now()
rocm_duringtime = rocm_endtime - rocm_starttime
print("%15s network rocm cost time is %s
s"%(child_model_network,rocm_duringtime))
opencl_starttime = datetime.datetime.now()
opencl_output = build("opencl",mod = mod,params = params,input_name
= model_network.input_name,input_data =
[GitHub] [tvm] wrongtest-intellif commented on pull request #13646: [Schedule][Bugfix] Fix decompose padding wrt the single child subtree
wrongtest-intellif commented on PR #13646: URL: https://github.com/apache/tvm/pull/13646#issuecomment-1366323467 cc @vinx13 @Hzfengsy @junrushao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627: URL: https://github.com/apache/tvm/pull/13627#discussion_r1058000486 ## gallery/how_to/work_with_microtvm/install_cmsis.rst: ## @@ -0,0 +1,30 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +..http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +Install CMSIS-NN Review Comment: This is not linked in the tutorial index and won't be found unless someone goes digging through the code. I'll add a comment though to clarify this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627: URL: https://github.com/apache/tvm/pull/13627#discussion_r1058000392 ## docs/conf.py: ## @@ -506,6 +650,12 @@ def process_docstring(app, what, name, obj, options, lines): from legacy_redirect import build_legacy_redirect +def strip_ipython_magic(app, docname, source): Review Comment: Yep - IPython magic is not always one word (e.g. `%%writefile output.txt`) but it is always exactly one line. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627:
URL: https://github.com/apache/tvm/pull/13627#discussion_r1057998312
##
docs/conf.py:
##
@@ -84,6 +86,148 @@ def git_describe_version(original_version):
version = git_describe_version(tvm.__version__)
release = version
+
+def monkey_patch(module_name, func_name):
+"""Helper function for monkey-patching library functions.
+
+Used to modify a few sphinx-gallery behaviors to make the "Open in Colab"
+button work correctly. Should be used as a decorator with arguments.
+"""
+module = import_module(module_name)
+original_func = getattr(module, func_name)
+
+def decorator(function):
+updated_func = partial(function, real_func=original_func)
+setattr(module, func_name, updated_func)
+return updated_func
+
+return decorator
+
+
+# This header replaces the default sphinx-gallery one in
sphinx_gallery/gen_rst.py.
+COLAB_HTML_HEADER = """
+.. DO NOT EDIT. THIS FILE WAS AUTOMATICALLY GENERATED BY
+.. TVM'S MONKEY-PATCHED VERSION OF SPHINX-GALLERY. TO MAKE
+.. CHANGES, EDIT THE SOURCE PYTHON FILE:
+.. "{0}"
+
+.. only:: html
+
+.. note::
+:class: sphx-glr-download-link-note
+
+This tutorial can be used interactively with Google Colab! You can
also click
+:ref:`here ` to run the Jupyter notebook
locally.
+
+.. image::
https://raw.githubusercontent.com/apache/web-data/main/images/utilities/colab_button.svg
+:align: center
+:target: {2}
+:width: 300px
+
+.. rst-class:: sphx-glr-example-title
+
+.. _sphx_glr_{1}:
+
+"""
+
+# Google Colab allows opening .ipynb files on GitHub by appending the GitHub
path to its own url.
+COLAB_URL_BASE = "https://colab.research.google.com/github;
+IPYTHON_GITHUB_BASE = "apache/tvm-site/blob/asf-site/docs/_downloads/"
+
+
+@monkey_patch("sphinx_gallery.gen_rst", "save_rst_example")
+def save_rst_example(example_rst, example_file, time_elapsed, memory_used,
gallery_conf, real_func):
+"""Monkey-patch save_rst_example to include the "Open in Colab" button."""
+
+# The url is the md5 hash of the notebook path.
+example_fname = os.path.relpath(example_file, gallery_conf["src_dir"])
+ref_fname = example_fname.replace(os.path.sep, "_")
+notebook_path = example_fname[:-2] + "ipynb"
+digest = md5(notebook_path.encode()).hexdigest()
+
+# Fixed documentation versions must link to different (earlier) .ipynb
notebooks.
+colab_url = f"{COLAB_URL_BASE}/{IPYTHON_GITHUB_BASE}"
+if "dev" not in version:
+colab_url += version + "/"
+colab_url += digest + "/" + os.path.basename(notebook_path)
+
+new_header = COLAB_HTML_HEADER.format(example_fname, ref_fname, colab_url)
+with patch("sphinx_gallery.gen_rst.EXAMPLE_HEADER", new_header):
+real_func(example_rst, example_file, time_elapsed, memory_used,
gallery_conf)
+
+
+@monkey_patch("sphinx_gallery.notebook", "rst2md")
+def rst2md(text, gallery_conf, target_dir, heading_levels, real_func):
+"""Monkey-patch rst2md to add limited include directive support to
sphinx-gallery."""
+
+include_re = re.compile(r"^([ \t]*)\.\. include::\s*(.+)\n", flags=re.M)
+
+def load_include(match):
+full_path = os.path.join(target_dir, match.group(2))
+with open(full_path) as f:
+lines = f.read()
+indented = textwrap.indent(lines, match.group(1)) + "\n"
+return indented
+
+text = re.sub(include_re, load_include, text)
+
+return real_func(text, gallery_conf, target_dir, heading_levels)
+
+
+INSTALL_TVM_DEV = f"""%%shell
+# Installs the latest dev build of TVM from PyPI. If you wish to build
+# from source, see https://tvm.apache.org/docs/install/from_source.html
+pip install apache-tvm --pre"""
+
+INSTALL_TVM_FIXED = f"""%%shell
+# Installs TVM version {version} from PyPI. If you wish to build
+# from source, see https://tvm.apache.org/docs/install/from_source.html
+pip install apache-tvm=={version}"""
+
+INSTALL_TVM_CUDA_DEV = f"""%%shell
+# Installs the latest dev build of TVM from PyPI, with CUDA enabled. To use
this,
+# you must request a Google Colab instance with a GPU by going to Runtime ->
+# Change runtime type -> Hardware accelerator -> GPU. If you wish to build from
+# source, see see https://tvm.apache.org/docs/install/from_source.html
+pip install tlcpack-nightly-cu113 --pre -f https://tlcpack.ai/wheels"";
+
+INSTALL_TVM_CUDA_FIXED = f"""%%shell
+# Installs TVM version {version} from PyPI, with CUDA enabled. To use this,
+# you must request a Google Colab instance with a GPU by going to Runtime ->
+# Change runtime type -> Hardware accelerator -> GPU. If you wish to build from
+# source, see see https://tvm.apache.org/docs/install/from_source.html
+pip install apache-tvm-cu113=={version} -f https://tlcpack.ai/wheels"";
Review Comment:
That would run fine, but I don't think that's as clear to read. I'd prefer
to keep explicitly specifying
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627: URL: https://github.com/apache/tvm/pull/13627#discussion_r1057997928 ## gallery/how_to/compile_models/from_coreml.py: ## @@ -23,13 +23,12 @@ This article is an introductory tutorial to deploy CoreML models with Relay. -For us to begin with, coremltools module is required to be installed. - -A quick solution is to install via pip +To begin, we must install coremltools: .. code-block:: bash -pip install -U coremltools --user Review Comment: `--user` works as you would expect in Colab. However, the other tutorials don't use that flag for their dependencies, so I changed this to be consistent. Might be out of scope - what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
masahi commented on code in PR #13642:
URL: https://github.com/apache/tvm/pull/13642#discussion_r1057990584
##
python/tvm/topi/x86/dense.py:
##
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
Review Comment:
This can be shared with the vnni compute.
##
python/tvm/topi/x86/dense.py:
##
@@ -373,6 +375,153 @@ def _callback(op):
return s
+def dense_amx_int8_compute(cfg, data, packed_w, bias=None):
+"""Compute for uint8 x int8 -> int32 dense"""
+m, k = data.shape
+n_o, _, n_i, _ = packed_w.shape
+ak = te.reduce_axis((0, k), name="k")
+
+C = te.compute(
+(m, n_o * n_i),
+lambda i, j: te.sum(
+data[i, ak].astype("int32")
+* packed_w[tvm.tir.indexdiv(j, 16), tvm.tir.indexdiv(ak, 4), j %
16, ak % 4].astype(
+"int32"
+),
+axis=ak,
+),
+tag="dense_amx_int8",
+attrs={"schedule_rule": "meta_schedule.dense_amx_int8"},
+)
+
+if bias is not None:
+C = te.compute(C.shape, lambda i, j: C[i, j] + bias[j],
tag=tag.BROADCAST)
+
+return C
+
+
+def dense_amx_int8_schedule(cfg, s, C, O, do_parallel=True):
+"""Schedule dense compute using AMX TMUL instruction"""
+# C: The output of GEMM
+# O: The output of the fused op
+def split_x(out):
+default_x_split_factor1 = 32
+default_x_split_factor2 = 2
+default_x_split_factor3 = 2
+default_x_split_factor4 = 2
+a_x = s[out].op.axis[-2]
+
+if cfg.is_fallback:
+a_xo, a_xi = s[out].split(a_x, factor=default_x_split_factor1)
+a_xo2, a_xo1 = s[out].split(a_xo, factor=default_x_split_factor2)
+a_xo3, a_xo2 = s[out].split(a_xo2, factor=default_x_split_factor3)
+a_xo4, a_xo3 = s[out].split(a_xo3, factor=default_x_split_factor4)
+return [a_xo4, a_xo3, a_xo2, a_xo1, a_xi]
+
+cfg.define_split("tile_x", a_x, num_outputs=5, filter=lambda x:
x.size[-1] == 32)
+return cfg["tile_x"].apply(s, out, a_x)
+
+def split_y(out):
+default_y_split_factor1 = 32
+default_y_split_factor2 = 4
+default_y_split_factor3 = 4
+default_y_split_factor4 = 4
+a_y = s[out].op.axis[-1]
+
+if cfg.is_fallback:
+a_yo1, a_yo = s[out].split(a_y, factor=default_y_split_factor1)
+a_yo2, a_yo1 = s[out].split(a_yo1, factor=default_y_split_factor2)
+a_yo3, a_yo2 = s[out].split(a_yo2, factor=default_y_split_factor3)
+a_yo4, a_yo3 = s[out].split(a_yo3, factor=default_y_split_factor4)
+return [a_yo4, a_yo3, a_yo2, a_yo1, a_yo]
+
+cfg.define_split("tile_y", a_y, num_outputs=5, filter=lambda y:
y.size[-1] == 32)
+return cfg["tile_y"].apply(s, out, a_y)
+
+def split_k(out, rd_axis):
+default_k_split_factor1 = 128
+default_k_split_factor2 = 2
+default_k_split_factor3 = 2
+default_k_split_factor4 = 2
+
+if cfg.is_fallback:
+a_ko, a_ki = s[out].split(rd_axis, factor=default_k_split_factor1)
+a_ko2, a_ko1 = s[out].split(a_ko, factor=default_k_split_factor2)
+a_ko3, a_ko2 = s[out].split(a_ko2, factor=default_k_split_factor3)
+a_ko4, a_ko3 = s[out].split(a_ko3, factor=default_k_split_factor4)
+return [a_ko4, a_ko3, a_ko2, a_ko1, a_ki]
+
+cfg.define_split("tile_k", rd_axis, num_outputs=5, filter=lambda y:
y.size[-1] == 128)
+return cfg["tile_k"].apply(s, out, rd_axis)
+
+a_x, a_y = C.op.axis
+(a_k,) = C.op.reduce_axis
+CF = s.cache_write(C, "amx.tmm")
+
+a_x3, a_x2, a_x1, a_xo, a_xi = split_x(C)
+a_y3, a_y2, a_y1, a_yo, a_yi = split_y(C)
+s[C].reorder(a_x3, a_y3, a_x2, a_y2, a_x1, a_y1, a_xo, a_yo, a_xi, a_yi)
+
+s[CF].compute_at(s[C], a_yo)
+
+(a_k_f,) = CF.op.reduce_axis
+a_x_f, a_y_f = CF.op.axis
+
+a_xo_f, a_xi_f = s[CF].split(a_x_f, factor=32)
+
+a_yo_f, a_yi_f = s[CF].split(a_y_f, factor=32)
+a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_ki_f = split_k(CF, a_k_f)
+s[CF].reorder(a_k3_f, a_k2_f, a_k1_f, a_ko_f, a_xo_f, a_yo_f, a_ki_f,
a_xi_f, a_yi_f)
+
+(m, k) = CF.op.input_tensors[0].shape
+(n, c, n_i, c_i) = CF.op.input_tensors[1].shape
+n = n * n_i
+
+s[CF].tensorize(a_ki_f, dot_32x128x32_u8s8s32_sapphirerapids(LDA=int(k)))
+s[C].tensorize(a_xi, acc_32x32_int32_sapphirerapids(LDC=int(n)))
+
+if C == O:
+fused = s[O].fuse(a_x3, a_y3)
+else:
+a_y3, a_y2, a_y1, a_yr, a_yi = split_y(O)
+a_x3, a_x2, a_x1, a_xr, a_xi = split_x(O)
+
+s[O].reorder(a_y3, a_x3, a_y2, a_x2, a_y1, a_x1, a_yr, a_xr, a_yi,
a_xi)
+s[O].vectorize(a_xi)
+
+fused = s[O].fuse(a_x3, a_y3)
+
+if do_parallel:
+s[O].parallel(fused)
+
+return s, fused
+
+
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627: URL: https://github.com/apache/tvm/pull/13627#discussion_r1057978156 ## docs/conf.py: ## @@ -84,6 +86,148 @@ def git_describe_version(original_version): version = git_describe_version(tvm.__version__) release = version + +def monkey_patch(module_name, func_name): Review Comment: Using stdlib functions is always good, but `unittest.mock.patch` does not quite fit this use case. I can't use `@patch` as a decorator, as the decorator must be used on the function **calling** the function we wish to monkey-patch (e.g. `save_rst_example`). I could just call `patch` as a function afterwards, but `patch` also does not include functionality for wrapping functions. It would be possible to make `patch()` work, but I think writing our own decorator is cleaner. I've clarified the comment to better explain this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
cbalint13 commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057955355 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: > > different clocking, timing & implementation on ASIC > > What kind of ASIC do you mean? * CPU, family of x86, different generations, varying extended ISA layouts: amx avx512 vnni avx2 ssse3 sse2 > > (auto)tensorization opportunities differ as inner loops match differently > Under `tensorization opportunities differ` do yo mean different number of lanes for different instruction set which can be reflected in potential different blocking size? * Yes, both input-widths and output-lanes yields different outcomes, varying performances. * E.g. autotensorizer will opportunistically search to permute & match inner loops to these varying sizes. > Or something else? * TVM is a compiler after all, to my knowledge the only capable of auto-tensorization with arbitrary intrinsic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
cbalint13 commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057955355 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: > > different clocking, timing & implementation on ASIC > > What kind of ASIC do you mean? * CPU, family of x86, different generations, varying extended ISA layouts: amx avx512 vnni avx2 ssse3 sse2 > > (auto)tensorization opportunities differ as inner loops match differently > Under `tensorization opportunities differ` do yo mean different number of lanes for different instruction set which can be reflected in potential different blocking size? * Yes, both input-widths and output-lanes yields different outcomes, varying performances. * E.g. autotensorizer will opportunistically search to permute & match inner loops to these varying sizes. > Or something else? * TVM is a compiler after all, to my knowledge the only one capable of auto-tensorization with arbitrary intrinsic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] elvin-n commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
elvin-n commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057947174 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: > different clocking, timing & implementation on ASIC What kind of ASIC do you mean? > (auto)tensorization opportunities differ as inner loops match differently Under `tensorization opportunities differ` do yo mean different number of lanes for different instruction set which can be reflected in potential different blocking size? Or something else? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
cbalint13 commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057926988 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: > The instruction set for SSE/AVX2/AVX512 for int8 is absolutely the same, the only difference is the number of lanes. Yes, same class doing integer dot products on immediate registers, but mention: - *different* clocking, timing & implementation on ASIC - (auto)tensorization opportunities differ as inner loops match *differently* > Additionally, the patterns how these int8 instructions (VPMADDUBSW/VPMADDWD/VPADDD) are used, is the >same as the only VNNI instruction (VPDPBUSD). Right. - VNNI insn. accumulates into int32 lanes in single step: vpdpbusd - AVX512 (incl. AVX2, SSSE3 ones) does same in two-step, e.g: pmaddubs + pmadd > I.e. it is reasonable to have the only tvm intrinsic, it is reasonable to remove VNNI from the name of the function, and it is reasonable to extend these intrinsic function to SSE and AVX2 that is not done yet in this PR - Indeed the proposed intrinsic merger is perfectly fine. - It was possible to question it with reasonable arguments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[tvm] branch main updated: [QNN] Change in Pass Context for lookup table calculation (#13660)
This is an automated email from the ASF dual-hosted git repository.
masahi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new f83055f90a [QNN] Change in Pass Context for lookup table calculation
(#13660)
f83055f90a is described below
commit f83055f90ad300fe60ad710260abba5a0693eefd
Author: ibsidorenko <98739392+ibsidore...@users.noreply.github.com>
AuthorDate: Wed Dec 28 00:10:15 2022 +0300
[QNN] Change in Pass Context for lookup table calculation (#13660)
Motivation:
It is possible to disable specific passes through the "disabled_pass"
parameter in the Pass Context. These "disabled" passes can be optional
for one target and mandatory for another one.
Since lookup table for some QNN operations (tanh, round and etc.) is
calculated on the host and some of disabled passes can be required for
the host, no need to disable these passes. This constant calculation/
evaluation is orthogonal to the compilation process for specific target.
What was changed:
This commit creates its own compilation Pass Context for lookup table
calculation and evaluation (for elemwise QNN ops: tanh, sqrt ...).
---
python/tvm/relay/qnn/op/canonicalizations.py | 23 +++
1 file changed, 19 insertions(+), 4 deletions(-)
diff --git a/python/tvm/relay/qnn/op/canonicalizations.py
b/python/tvm/relay/qnn/op/canonicalizations.py
index 1f2c57c6da..6bfcd34aba 100644
--- a/python/tvm/relay/qnn/op/canonicalizations.py
+++ b/python/tvm/relay/qnn/op/canonicalizations.py
@@ -23,10 +23,25 @@ from tvm import relay
def run_const_expr(expr: "relay.Expr") -> np.ndarray:
-"""Evaluate a const expression, receiving result as np array."""
-mod = tvm.IRModule.from_expr(expr)
-vm_exe = relay.create_executor("vm", mod=mod)
-return vm_exe.evaluate()().asnumpy()
+"""Evaluate a const expression, receiving result as np array.
+
+If a number of passes are disabled in the current Pass Context, then there
is no need to disable
+these passes for const expression evaluation as well. That's why we use
empty list
+"disabled_pass=[]", all other arguments are inherited from the current
Pass Context.
+"""
+curr_pass_ctx = tvm.ir.transform.PassContext.current()
+with tvm.ir.transform.PassContext(
+opt_level=curr_pass_ctx.opt_level,
+required_pass=curr_pass_ctx.required_pass,
+disabled_pass=[],
+instruments=curr_pass_ctx.instruments,
+config=curr_pass_ctx.config,
+):
+mod = tvm.IRModule.from_expr(expr)
+vm_exe = relay.create_executor("vm", mod=mod)
+output = vm_exe.evaluate()().asnumpy()
+
+return output
def create_integer_lookup_table(
[GitHub] [tvm] masahi merged pull request #13660: [QNN] Change in Pass Context for lookup table calculation
masahi merged PR #13660: URL: https://github.com/apache/tvm/pull/13660 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] masahi commented on a diff in pull request #13660: [QNN] Change in Pass Context for lookup table calculation
masahi commented on code in PR #13660:
URL: https://github.com/apache/tvm/pull/13660#discussion_r1057915918
##
python/tvm/relay/qnn/op/canonicalizations.py:
##
@@ -23,10 +23,25 @@
def run_const_expr(expr: "relay.Expr") -> np.ndarray:
-"""Evaluate a const expression, receiving result as np array."""
-mod = tvm.IRModule.from_expr(expr)
-vm_exe = relay.create_executor("vm", mod=mod)
-return vm_exe.evaluate()().asnumpy()
+"""Evaluate a const expression, receiving result as np array.
+
+If a number of passes are disabled in the current Pass Context, then there
is no need to disable
+these passes for const expression evaluation as well. That's why we use
empty list
+"disabled_pass=[]", all other arguments are inherited from the current
Pass Context.
Review Comment:
I see, you do want to enable QNN legalization (which is disabled in the
original ctx) here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
masahi commented on issue #13664: URL: https://github.com/apache/tvm/issues/13664#issuecomment-1366180454 You can try the latest main, 520f2c594. But again, the specific version doesn't matter, HF has always been supported. I don't know what is "Nimble functions". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] guberti commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
guberti commented on code in PR #13627:
URL: https://github.com/apache/tvm/pull/13627#discussion_r1057909768
##
apps/microtvm/pyproject.toml:
##
@@ -98,22 +95,23 @@ onnxoptimizer = { version = "==0.2.6", optional = true }
onnxruntime = { version = "==1.9.0", optional = true }
# Pytorch (also used by ONNX)
-torch = { version = "==1.11.0", optional = true }
-torchvision = { version = "==0.12.0", optional = true }
+torch = { version = "==1.11.0" }
Review Comment:
cc @mehrdadh who made this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tvm] elvin-n commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
elvin-n commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057896792 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: The instruction set for SSE/AVX2/AVX512 for int8 is absolutely the same, the only difference is the number of lanes. Additionally, the patterns how these int8 instructions (VPMADDUBSW/VPMADDWD/VPADDD) are used, is the same as the only VNNI instruction (VPDPBUSD). I.e. it is reasonable to have the only tvm intrinsic, it is reasonable to remove VNNI from the name of the function, and it is reasonable to extend these intrinsic function to SSE and AVX2 that is not done yet in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[tvm] branch main updated: [Relay][Frontend] Span filling common API (#13402)
This is an automated email from the ASF dual-hosted git repository.
areusch pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tvm.git
The following commit(s) were added to refs/heads/main by this push:
new 520f2c594b [Relay][Frontend] Span filling common API (#13402)
520f2c594b is described below
commit 520f2c594b3a4dde60ebdb10de448112130ab38a
Author: Chun-I Tsai
AuthorDate: Wed Dec 28 02:37:51 2022 +0800
[Relay][Frontend] Span filling common API (#13402)
- Expose and add span attribute of Expr-derived types from C++ to Python
- Add common API of span filling
- Add test cases of span filling
- Add function to control whether to fill span via environment variable
- Modify the way of pretty-print to print span
Co-authored-by: Joey Tsai
---
python/tvm/relay/expr.py | 202 +++
python/tvm/relay/frontend/common.py | 165 +++-
python/tvm/relay/function.py | 7 +-
python/tvm/relay/loops.py| 2 +-
python/tvm/testing/utils.py | 22
src/ir/span.cc | 4 +
src/relay/ir/expr.cc | 88 ---
src/relay/ir/function.cc | 4 +-
tests/python/frontend/test_common.py | 194 -
tests/python/relay/utils/tag_span.py | 108 +++
10 files changed, 750 insertions(+), 46 deletions(-)
diff --git a/python/tvm/relay/expr.py b/python/tvm/relay/expr.py
index fefc285723..88b84bbe7e 100644
--- a/python/tvm/relay/expr.py
+++ b/python/tvm/relay/expr.py
@@ -171,10 +171,28 @@ class Constant(ExprWithOp):
--
data : tvm.nd.NDArray
The data content of the constant expression.
+
+span: Optional[tvm.relay.Span]
+Span that points to original source code.
"""
-def __init__(self, data):
-self.__init_handle_by_constructor__(_ffi_api.Constant, data)
+def __init__(self, data, span=None):
+self.__init_handle_by_constructor__(_ffi_api.Constant, data, span)
+
+
+@tvm._ffi.register_func("relay.ConstantWithFields")
+def ConstantWithFields(
+constant,
+data=None,
+virtual_device=None,
+span=None,
+):
+"""
+Returns constant with the given properties. A None property denotes 'no
change'.
+Returns constant if all properties are unchanged. Otherwise, returns a
copy with the new
+fields.
+"""
+return _ffi_api.ConstantWithFields(constant, data, virtual_device, span)
@tvm._ffi.register_object("relay.Tuple")
@@ -187,7 +205,7 @@ class Tuple(ExprWithOp):
The fields in the tuple.
span: Optional[tvm.relay.Span]
-Span that points to original source code
+Span that points to original source code.
"""
def __init__(self, fields, span=None):
@@ -205,6 +223,16 @@ class Tuple(ExprWithOp):
raise TypeError("astype cannot be used on tuple")
+@tvm._ffi.register_func("relay.TupleWithFields")
+def TupleWithFields(tup, fields=None, virtual_device=None, span=None):
+"""
+Returns tuple with the given properties. A None property denotes 'no
change'.
+Returns tuple if all properties are unchanged. Otherwise, returns a copy
with the new
+fields.
+"""
+return _ffi_api.TupleWithFields(tup, fields, virtual_device, span)
+
+
@tvm._ffi.register_object("relay.Var")
class Var(ExprWithOp):
"""A local variable in Relay.
@@ -221,10 +249,13 @@ class Var(ExprWithOp):
type_annotation: tvm.relay.Type, optional
The type annotation on the variable.
+
+span: Optional[tvm.relay.Span]
+Span that points to original source code.
"""
-def __init__(self, name_hint, type_annotation=None):
-self.__init_handle_by_constructor__(_ffi_api.Var, name_hint,
type_annotation)
+def __init__(self, name_hint, type_annotation=None, span=None):
+self.__init_handle_by_constructor__(_ffi_api.Var, name_hint,
type_annotation, span)
@property
def name_hint(self):
@@ -233,6 +264,16 @@ class Var(ExprWithOp):
return name
+@tvm._ffi.register_func("relay.VarWithFields")
+def VarWithFields(variable, vid=None, type_annotation=None,
virtual_device=None, span=None):
+"""
+Returns var with the given properties. A None property denotes 'no change'.
+Returns var if all properties are unchanged. Otherwise, returns a copy
with the new
+fields.
+"""
+return _ffi_api.VarWithFields(variable, vid, type_annotation,
virtual_device, span)
+
+
@tvm._ffi.register_object("relay.Call")
class Call(ExprWithOp):
"""Function call node in Relay.
@@ -256,7 +297,7 @@ class Call(ExprWithOp):
used in advanced usecase of template functions.
span: Optional[tvm.relay.Span]
-Span that points to original source code
+Span that points to original source code.
"""
def __init__(self, op, args, attrs=None,
[GitHub] [tvm] areusch merged pull request #13402: [Relay][Frontend] Span filling common API
areusch merged PR #13402: URL: https://github.com/apache/tvm/pull/13402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] areusch commented on a diff in pull request #13627: [docs] Add "Open with Colab" button to documentation
areusch commented on code in PR #13627:
URL: https://github.com/apache/tvm/pull/13627#discussion_r1057808388
##
docs/conf.py:
##
@@ -84,6 +86,148 @@ def git_describe_version(original_version):
version = git_describe_version(tvm.__version__)
release = version
+
+def monkey_patch(module_name, func_name):
Review Comment:
you might consider using `unittest.mock.patch` here since it's in the stdlib
##
docs/conf.py:
##
@@ -506,6 +650,12 @@ def process_docstring(app, what, name, obj, options,
lines):
from legacy_redirect import build_legacy_redirect
+def strip_ipython_magic(app, docname, source):
Review Comment:
add a docstring comment just to clarify the function. should this be
removing the whole line?
##
gallery/how_to/work_with_microtvm/micro_tflite.py:
##
@@ -26,109 +26,36 @@
"""
##
-# .. note::
-# If you want to run this tutorial on the microTVM Reference VM, download
the Jupyter
-# notebook using the link at the bottom of this page and save it into the
TVM directory. Then:
#
-# #. Login to the reference VM with a modified ``vagrant ssh`` command:
+# .. include::
../../../../gallery/how_to/work_with_microtvm/install_dependencies.rst
#
-# ``$ vagrant ssh -- -L:localhost:``
-#
-# #. Install jupyter: ``pip install jupyterlab``
-# #. ``cd`` to the TVM directory.
-# #. Install tflite: poetry install -E importer-tflite
-# #. Launch Jupyter Notebook: ``jupyter notebook``
-# #. Copy the localhost URL displayed, and paste it into your browser.
-# #. Navigate to saved Jupyter Notebook (``.ipynb`` file).
-#
-#
-# Setup
-# -
-#
-# Install TFLite
-# ^^
-#
-# To get started, TFLite package needs to be installed as prerequisite. You
can do this in two ways:
-#
-# 1. Install tflite with ``pip``
-#
-# .. code-block:: bash
-#
-# pip install tflite=2.1.0 --user
-#
-# 2. Generate the TFLite package yourself. The steps are the following:
-#
-# Get the flatc compiler.
-# Please refer to https://github.com/google/flatbuffers for details
-# and make sure it is properly installed.
-#
-# .. code-block:: bash
-#
-# flatc --version
-#
-# Get the TFLite schema.
-#
-# .. code-block:: bash
-#
-# wget
https://raw.githubusercontent.com/tensorflow/tensorflow/r1.13/tensorflow/lite/schema/schema.fbs
-#
-# Generate TFLite package.
-#
-# .. code-block:: bash
-#
-# flatc --python schema.fbs
-#
-# Add the current folder (which contains generated tflite module) to
PYTHONPATH.
-#
-# .. code-block:: bash
-#
-# export PYTHONPATH=${PYTHONPATH:+$PYTHONPATH:}$(pwd)
-#
-# To validate that the TFLite package was installed successfully, ``python -c
"import tflite"``
-#
-# Install Zephyr (physical hardware only)
-# ^^^
-#
-# When running this tutorial with a host simulation (the default), you can use
the host ``gcc`` to
-# build a firmware image that simulates the device. When compiling to run on
physical hardware, you
-# need to install a *toolchain* plus some target-specific dependencies.
microTVM allows you to
-# supply any compiler and runtime that can launch the TVM RPC server, but to
get started, this
-# tutorial relies on the Zephyr RTOS to provide these pieces.
-#
-# You can install Zephyr by following the
-# `Installation Instructions
`_.
-#
-# Aside: Recreating your own Pre-Trained TFLite model
-# The tutorial downloads a pretrained TFLite model. When working with
microcontrollers
-# you need to be mindful these are highly resource constrained devices as
such standard
-# models like MobileNet may not fit into their modest memory.
-#
-# For this tutorial, we'll make use of one of the TF Micro example models.
-#
-# If you wish to replicate the training steps see:
-#
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/micro/examples/hello_world/train
-#
-#.. note::
-#
-# If you accidentally download the example pretrained model from:
-#
-# ``wget
https://storage.googleapis.com/download.tensorflow.org/models/tflite/micro/hello_world_2020_04_13.zip``
-#
-# this will fail due to an unimplemented opcode (114)
-#
-# Load and prepare the Pre-Trained Model
-# --
-#
-# Load the pretrained TFLite model from a file in your current
-# directory into a buffer
# sphinx_gallery_start_ignore
from tvm import testing
testing.utils.install_request_hook(depth=3)
# sphinx_gallery_end_ignore
+# You can skip the following two sections (installing Zephyr and CMSIS-NN) if
the following flag is False.
Review Comment:
could you explain why maybe?
##
docs/conf.py:
##
@@ -84,6 +86,148 @@ def git_describe_version(original_version):
version =
[tvm] branch dependabot/pip/docker/python/setuptools-65.5.1 created (now e53ab43cf2)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/docker/python/setuptools-65.5.1 in repository https://gitbox.apache.org/repos/asf/tvm.git at e53ab43cf2 Bump setuptools from 62.3.2 to 65.5.1 in /docker/python No new revisions were added by this update.
[GitHub] [tvm] tvm-bot commented on pull request #13665: Bump setuptools from 62.3.2 to 65.5.1 in /docker/python
tvm-bot commented on PR #13665: URL: https://github.com/apache/tvm/pull/13665#issuecomment-1366040353 Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment. * No users to tag found in teams: `type:dependency-security`, `python` See [#10317](https://github.com/apache/tvm/issues/10317) for details Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] dependabot[bot] opened a new pull request, #13665: Bump setuptools from 62.3.2 to 65.5.1 in /docker/python
dependabot[bot] opened a new pull request, #13665: URL: https://github.com/apache/tvm/pull/13665 Bumps [setuptools](https://github.com/pypa/setuptools) from 62.3.2 to 65.5.1. Release notes Sourced from https://github.com/pypa/setuptools/releases;>setuptools's releases. v65.5.1 No release notes provided. v65.5.0 No release notes provided. v65.4.1 No release notes provided. v65.4.0 No release notes provided. v65.3.0 No release notes provided. v65.2.0 No release notes provided. v65.1.1 No release notes provided. v65.1.0 No release notes provided. v65.0.2 No release notes provided. v65.0.1 No release notes provided. v65.0.0 No release notes provided. v64.0.3 No release notes provided. v64.0.2 No release notes provided. v64.0.1 No release notes provided. v64.0.0 No release notes provided. v63.4.3 No release notes provided. v63.4.2 No release notes provided. ... (truncated) Changelog Sourced from https://github.com/pypa/setuptools/blob/main/CHANGES.rst;>setuptools's changelog. v65.5.1 Misc https://github-redirect.dependabot.com/pypa/setuptools/issues/3638;>#3638: Drop a test dependency on the mock package, always use :external+python:py:mod:unittest.mock -- by :user:hroncok https://github-redirect.dependabot.com/pypa/setuptools/issues/3659;>#3659: Fixed REDoS vector in package_index. v65.5.0 Changes ^^^ https://github-redirect.dependabot.com/pypa/setuptools/issues/3624;>#3624: Fixed editable install for multi-module/no-package src-layout projects. https://github-redirect.dependabot.com/pypa/setuptools/issues/3626;>#3626: Minor refactorings to support distutils using stdlib logging module. Documentation changes ^ https://github-redirect.dependabot.com/pypa/setuptools/issues/3419;>#3419: Updated the example version numbers to be compliant with PEP-440 on the Specifying Your Project’s Version page of the user guide. Misc https://github-redirect.dependabot.com/pypa/setuptools/issues/3569;>#3569: Improved information about conflicting entries in the current working directory and editable install (in documentation and as an informational warning). https://github-redirect.dependabot.com/pypa/setuptools/issues/3576;>#3576: Updated version of validate_pyproject. v65.4.1 Misc https://github-redirect.dependabot.com/pypa/setuptools/issues/3613;>#3613: Fixed encoding errors in expand.StaticModule when system default encoding doesn't match expectations for source files. https://github-redirect.dependabot.com/pypa/setuptools/issues/3617;>#3617: Merge with pypa/distutils@6852b20 including fix for https://github-redirect.dependabot.com/pypa/distutils/issues/181;>pypa/distutils#181. v65.4.0 Changes ^^^ https://github-redirect.dependabot.com/pypa/setuptools/issues/3609;>#3609: Merge with pypa/distutils@d82d926 including support for DIST_EXTRA_CONFIG in https://github-redirect.dependabot.com/pypa/distutils/issues/177;>pypa/distutils#177. v65.3.0 ... (truncated) Commits https://github.com/pypa/setuptools/commit/a462cb5edb324dcc56f903524b742305e4087014;>a462cb5 Bump version: 65.5.0 → 65.5.1 https://github.com/pypa/setuptools/commit/de35d8be997c9f8508b425e33a1b6e52431091fa;>de35d8b Merge pull request https://github-redirect.dependabot.com/pypa/setuptools/issues/3656;>#3656 from bmorris3/typos https://github.com/pypa/setuptools/commit/58e23de0d4c2ce0f2502d072bb9a2ed1e2ab0ba6;>58e23de Update changelog. Ref https://github-redirect.dependabot.com/pypa/setuptools/issues/3659;>#3659. https://github.com/pypa/setuptools/commit/43a9c9bfa6aa626ec2a22540bea28d2ca77964be;>43a9c9b Limit the amount of whitespace to search/backtrack. Fixes https://github-redirect.dependabot.com/pypa/setuptools/issues/3659;>#3659. https://github.com/pypa/setuptools/commit/579134321d4d9397c886a5cb50cc26d0e3fa4279;>5791343 Add test capturing failed expectation. Ref https://github-redirect.dependabot.com/pypa/setuptools/issues/3659;>#3659. https://github.com/pypa/setuptools/commit/1f97905bc40310ca454ff1ea3884f233b7dcc88c;>1f97905 ⚫ Fade to black. https://github.com/pypa/setuptools/commit/6254567c6ae323bb8ce19a6930ae3cc5f7fb25cc;>6254567 Remove workaround for emacs. https://github.com/pypa/setuptools/commit/729b180e926634930c21ccce5558780d42707763;>729b180 ⚫ Fade to black. https://github.com/pypa/setuptools/commit/c068081a7234a0c5c322a9312654e7d0f4aaa8d1;>c068081 Typo corrections https://github.com/pypa/setuptools/commit/f777a40ed9abf529906c2939f80a184a5ed035fa;>f777a40 Suppress deprecation warning in --rsyncdir. Workaround for https://github-redirect.dependabot.com/pypa/setuptools/issues/3655;>#3655. Additional commits viewable in
[GitHub] [tvm] JamesTheZ commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
JamesTheZ commented on issue #13664: URL: https://github.com/apache/tvm/issues/13664#issuecomment-1366010972 @masahi What is the tvm commit id you are using? I have tried to rebuild TVM and use the same software version as you, but the problem remains. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Qianshui-Jiang commented on PR #13642: URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365954741 @masahi, BTW, SDE would be very slow, the repeat times of time_evaluator should be small. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] Qianshui-Jiang commented on pull request #13642: [Tensorize][runtime] Add support for AMX(Advanced Matrix Extensions) through Tensor intrinsics
Qianshui-Jiang commented on PR #13642: URL: https://github.com/apache/tvm/pull/13642#issuecomment-1365952931 @masahi Sure, you can use SDE by assign platform info like `-spr`, But if you are not built on 5.16 kernel, you have to comment all the system related code `runtime.amx_init` , and don't need to init amx before the testcase when using SDE; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] ibsidorenko commented on a diff in pull request #13660: [QNN] Change in Pass Context for lookup table calculation
ibsidorenko commented on code in PR #13660:
URL: https://github.com/apache/tvm/pull/13660#discussion_r1057708382
##
python/tvm/relay/qnn/op/canonicalizations.py:
##
@@ -23,10 +23,25 @@
def run_const_expr(expr: "relay.Expr") -> np.ndarray:
-"""Evaluate a const expression, receiving result as np array."""
-mod = tvm.IRModule.from_expr(expr)
-vm_exe = relay.create_executor("vm", mod=mod)
-return vm_exe.evaluate()().asnumpy()
+"""Evaluate a const expression, receiving result as np array.
+
+If a number of passes are disabled in the current Pass Context, then there
is no need to disable
+these passes for const expression evaluation as well. That's why we use
empty list
+"disabled_pass=[]", all other arguments are inherited from the current
Pass Context.
Review Comment:
I put here `disabled_pass=[]` to ignore the original `disabled_pass` list
from the current context. Otherwise VM inherits non-empty `disabled_pass` list
and fails.
Note, we call this code only for several qnn ops: tanh, sqrt, erf, exp +
some other ops. It does not affect global constant folding.
For clarification, here is my example to illustrate this issue:
I am trying to compile subgraph with `qnn.tanh` operation for Hexagon target
**without** QNN canonicalization but **with** QNN legalization (this is
acceptable for Hexagon). For `qnn.tanh` we compose lookup table by means of
Virtual Machine and compute it on the host. For most of users/developers host
is x86 cpu. But for x86 QNN canonicalization is mandatory pass, otherwise it
fails.
Looks like the simplest way to fix this issue is to create new Pass Context
and ignore the original `disabled_pass` list for lookup table calculation and
constant evaluation. Other arguments (opt_level, instruments, config ...) I
inherit from the current context.
P.S. Corresponding unit test will be added in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tvm] JamesTheZ commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
JamesTheZ commented on issue #13664: URL: https://github.com/apache/tvm/issues/13664#issuecomment-1365905766 > How can the Nimble functions be evaluated if the dynamic shape is not fully supported for torch frontends? Are there benchmarks for Nimble? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] cbalint13 commented on a diff in pull request #13621: WIP: [TIR][TOPI][x86][CI] Support skylake avx512
cbalint13 commented on code in PR #13621: URL: https://github.com/apache/tvm/pull/13621#discussion_r1057322567 ## python/tvm/relay/op/strategy/x86.py: ## @@ -627,16 +627,16 @@ def batch_matmul_strategy_cpu(attrs, inputs, out_type, target): if ( not attrs.transpose_a and attrs.transpose_b -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) and inputs[0].dtype == "uint8" and inputs[1].dtype == "int8" and inputs[1].shape[-2] % 16 == 0 and inputs[1].shape[-1] % 4 == 0 ): strategy.add_implementation( -wrap_compute_batch_matmul(topi.x86.batch_matmul_vnni_compute, need_out_dtype=True), -wrap_topi_schedule(topi.x86.schedule_batch_matmul_vnni), -name="batch_matmul_vnni.x86", Review Comment: > Hello @cbalint13! Your view looks reasonable and there is no problems to reimplement it from my side. But I did not implement method `dot_16x1x16_uint8_int8_int32` with conditions on tensor_intrin.py side and thought that it is brick to build some concept. @elvin-n and @jwfromm what do you think about Balint's view? @vvchernov , Thanks for clarifications, I see your point, it is perfectly fine way too. I think by making all CI tests to pass in green more reviewers will come. --- I try sum up, on this very pinned PR change on **strategy/x86.py**, visibile on top of this thread: ``` -from tvm.topi.x86.utils import target_has_vnni +from tvm.topi.x86.utils import target_has_avx512 -and target_has_vnni(mcpu) +and target_has_avx512(mcpu) -wrap_compute_dense(topi.x86.dense_vnni), -wrap_topi_schedule(topi.x86.schedule_dense_vnni), +wrap_compute_dense(topi.x86.dense_int8), +wrap_topi_schedule(topi.x86.schedule_dense_int8), ``` * This merge vnni to avx512 (under new **dense_int8** umbrella) arguing that VNNI is subset of AVX512 group. * VNNI is subset of AVX512 group, however there are CPU having AVX512 but **no VNNI** [1]. [1] https://en.wikipedia.org/wiki/AVX-512#VNNI My view was to leave separate avx512 & vnni(as was) in strategy/x86.py (not to merge vnni->avx512) My argument was to triage any SIMD right in **strategy/x86.py** as upcoming AMX do [here](https://github.com/apache/tvm/blob/dd1eb2498a4e2c867a16d7d9ccea010c396e10ae/python/tvm/relay/op/strategy/x86.py#L594-L621) + plevel control. I saw VNNI and AVX512 +(AVX2, SSSE3) as potentialy independend things, moreover choosable via "llvm +mattr=...". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] masahi closed issue #13664: [Bug] Fail to convert standard huggingface BERT model
masahi closed issue #13664: [Bug] Fail to convert standard huggingface BERT model URL: https://github.com/apache/tvm/issues/13664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] masahi commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
masahi commented on issue #13664: URL: https://github.com/apache/tvm/issues/13664#issuecomment-1365793554 Mine is ``` torch version: 1.13.0+cpu, tvm version: 0.11.dev0, transformers version: 4.24.0 ``` but Huggingface BERT should be supported regardless of the PT or `transformers` versions. I have the script https://github.com/masahi/torchscript-to-tvm/blob/master/transformers/bert_clean.py working for a long time. > By the way, does TVM support converting dynamic-shaped models currently? Some frontends (ONNX, maybe TF too) support dynamic input shape. The PT frontend doesn't support dynamic input shape, but it supports dynamic-shape output (MaskRCNN etc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] JamesTheZ commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
JamesTheZ commented on issue #13664: URL: https://github.com/apache/tvm/issues/13664#issuecomment-1365750370 By the way, does TVM support converting dynamic-shaped models currently? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tvm] JamesTheZ commented on issue #13664: [Bug] Fail to convert standard huggingface BERT model
JamesTheZ commented on issue #13664:
URL: https://github.com/apache/tvm/issues/13664#issuecomment-1365749841
> It looks like amp is adding some dynamism, do you know why?
>
> The following modified script that removed amp autocast works:
>
> ```
> from transformers import BertTokenizer, BertConfig, BertModel
> import tvm
> from tvm import relay
> import torch
>
>
> class AutoModelAMP(BertModel):
> def __init__(self, *args, **keyargs):
> super().__init__(*args, **keyargs)
>
> def forward(self, *args, **keyargs):
> return super().forward(*args, **keyargs)
>
>
> config = BertConfig.from_pretrained("bert-large-uncased")
> config.return_dict = False
> config.torchscript = True
> model = AutoModelAMP(config).eval()
>
> batch = 1
> seq_len = 128
> input = torch.zeros([batch, seq_len], dtype=torch.int).long()
> inputs = {
> "input_ids": input,
> "attention_mask": input,
> "token_type_ids": input,
> }
> input_list = []
> for k, v in inputs.items():
> input_list.append(v)
>
> shape_list = []
> for k, v in inputs.items():
> shape_list.append((k, v.shape))
>
> traced_model = torch.jit.trace(
> model, input_list, strict=False).eval()
> mod, params = relay.frontend.from_pytorch(traced_model, shape_list)
> ```
Using the above script without autocast still results in the same error to
me. The environment by executing `python -c 'import torch, tvm, transformers;
print(f"torch version: {torch.__version__}, tvm version: {tvm.__version__},
transformers version: {transformers.__version__}");'` is:
```
torch version: 1.12.0+cu113, tvm version: 0.11.dev0, transformers version:
4.25.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@tvm.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
