Lunderberg commented on a change in pull request #10538:
URL: https://github.com/apache/tvm/pull/10538#discussion_r826146439
##########
File path: python/tvm/tir/function.py
##########
@@ -239,3 +240,40 @@ def get(name: str):
The TensorIntrin with the specified name.
"""
return _ffi_api.TensorIntrinGet(name) # pylint: type: ignore
+
+
+@tvm._ffi.register_object("tir.IndexMap")
+class IndexMap(Object):
+ """A mapping from multi-dimensional indices to another set of
multi-dimensional indices
+
+ Parameters
+ ----------
+ initial_indices : List[Var]
+ Variables representing the indices prior to remapping.
+ final_indices : List[PrimExpr]
+ Expressions defining the indices after remapping.
+ """
+
+ initial_indices: List[Var]
+ final_indices: List[PrimExpr]
+
+ @staticmethod
+ def from_func(func: Callable) -> "IndexMap":
Review comment:
It looks like this conversation isn't quite as flexible as the version
used by `te.Stage.transform_layout`. The main differences are (1) the
parameter names of `func` aren't propagated into the `Var` names, and (2)
`*args` are not handled.
For (1), can we propagate the parameter names?
For (2), I don't think we can correctly handle this case, since it requires
knowing how many buffer dimensions should be expanded into the `*args`. Can we
throw an error if `func` has arguments that can't be correctly handled?
(Overly-dense one-liner: `assert all(param.kind in
[inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD]
for param in inspect.signature(func).parameters.values()`)
##########
File path: src/tir/ir/index_map.cc
##########
@@ -142,6 +152,40 @@ Array<PrimExpr> IndexMapNode::MapShape(const
Array<PrimExpr>& shape) const {
return output;
}
+String IndexMapNode::ToPythonString() const {
Review comment:
I really like this function for making sure that duplicate `Var` have
human-readable names. Can we use it in the `ReprPrinter` as well?
##########
File path: python/tvm/tir/schedule/schedule.py
##########
@@ -2111,6 +2112,82 @@ def after_unannotate(a: T.handle, b: T.handle) -> None:
self, block_or_loop, ann_key
)
+ ########## Schedule: Layout transformation ##########
+
+ @type_checked
+ def transform_layout(
+ self,
+ block: BlockRV,
+ buffer_index: int,
+ is_write_index: bool,
+ index_map: Union[IndexMap, Callable],
+ ) -> None:
+ """Apply a transformation represented by IndexMap to buffer
+ Parameters
+ ----------
+ block_rv : BlockRV
+ The block that accesses the target buffer
+ buffer_index: int
+ The index of the buffer in block's read or write region
+ is_write_index : bool
+ Whether the buffer_index is the index of the block's write region
+ index_map : Union[IndexMap, Callable]
+ The transformation to apply
+
+ Examples
+ --------
+ Before transform_layout, in TensorIR, the IR is:
+
+ .. code-block:: python
+
+ @T.prim_func
+ def before_transform_layout(a: T.handle, c: T.handle) -> None:
+ A = T.match_buffer(a, (128, 128), "float32")
+ B = T.alloc_buffer((128, 128), "float32")
+ C = T.match_buffer(c, (128, 128), "float32")
+ for i, j in T.grid(128, 128):
+ with T.block("B"):
+ vi, vj = T.axis.remap("SS", [i, j])
+ B[vi, vj] = A[vi, vj] * 2.0
+ for i, j in T.grid(128, 128):
+ with T.block("C"):
+ vi, vj = T.axis.remap("SS", [i, j])
+ C[vi, vj] = B[vi, vj] + 1.0
+
+ Create the schedule and do transform_layout:
+
+ .. code-block:: python
+
+ sch = tir.Schedule(before_storage_align)
+ sch.transform_layout(sch.get_block("B"), buffer_index=0,
is_write_index=True,
+ index_map=lambda m, n: (m // 16, n // 16, m %
16, n % 16))
+ print(sch.mod["main"].script())
+
+ After applying transform_layout, the IR becomes:
+
+ .. code-block:: python
+
+ @T.prim_func
+ def two_elementwise_transformed_intermediate_buffer(a: T.handle,
c: T.handle) -> None:
+ A = T.match_buffer(a, (128, 128), "float32")
+ B = T.alloc_buffer((8, 8, 16, 16), "float32")
+ C = T.match_buffer(c, (128, 128), "float32")
+ for i, j in T.grid(128, 128):
+ with T.block("B"):
+ vi, vj = T.axis.remap("SS", [i, j])
+ B[vi // 16, vj // 16, vi % 16, vj % 16] = A[vi, vj] *
2.0
+ for i, j in T.grid(128, 128):
+ with T.block("C"):
+ vi, vj = T.axis.remap("SS", [i, j])
+ C[vi, vj] = B[vi // 16, vj // 16, vi % 16, vj % 16] +
1.0
+
+ """
+ if callable(index_map):
+ index_map = IndexMap.from_func(index_map)
Review comment:
Related to the previous comment, since we have access to the buffer
itself here, can we support `*args` in this code path?
##########
File path: src/tir/schedule/primitive/layout_transformation.cc
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "../utils.h"
+
+namespace tvm {
+namespace tir {
+
+class TransformLayoutRewriter : private StmtExprMutator {
+ public:
+ /*!
+ * \brief Rewrite the access to the buffer after the transformation
+ * \param scope_stmt The parent statement that contains all accesses to the
target buffer
+ * \param old_buffer The target buffer before transformation
+ * \param new_buffer The new buffer after transformation
+ * \param index_map The transformation applied to the buffer
+ * \return The new AST rooting at the original parent scope and the map from
the old block to the
+ * new block
+ */
+ static std::pair<Stmt, Map<Block, Block>> Rewrite(const Stmt& scope_stmt,
+ const Buffer& old_buffer,
+ const Buffer& new_buffer,
+ const IndexMap& index_map)
{
+ TransformLayoutRewriter rewriter(old_buffer, new_buffer, index_map);
+ Stmt result = rewriter(scope_stmt);
+ return {result, rewriter.block_sref_reuse_};
+ }
+
+ private:
+ TransformLayoutRewriter(const Buffer& old_buffer, const Buffer& new_buffer,
+ const IndexMap& index_map)
+ : old_buffer_(old_buffer),
+ new_buffer_(new_buffer),
+ index_map_(index_map),
+ buffer_data_to_buffer_{{new_buffer->data, new_buffer}} {}
+
+ void RewriteBufferAccess(Buffer* buffer, Array<PrimExpr>* indices) {
+ *buffer = new_buffer_;
+ *indices = index_map_->MapIndices(*indices);
+ }
+
+ PrimExpr VisitExpr_(const BufferLoadNode* op) final {
+ BufferLoad buffer_load =
Downcast<BufferLoad>(StmtExprMutator::VisitExpr_(op));
+ if (buffer_load->buffer.same_as(old_buffer_)) {
+ auto* n = buffer_load.CopyOnWrite();
+ RewriteBufferAccess(&n->buffer, &n->indices);
+ }
+ return std::move(buffer_load);
+ }
+
+ Stmt VisitStmt_(const BufferStoreNode* op) final {
+ BufferStore buffer_store =
Downcast<BufferStore>(StmtExprMutator::VisitStmt_(op));
+ if (buffer_store->buffer.same_as(old_buffer_)) {
+ auto* n = buffer_store.CopyOnWrite();
+ RewriteBufferAccess(&n->buffer, &n->indices);
+ }
+ return std::move(buffer_store);
+ }
+
+ void RewriteAccessRegion(Array<BufferRegion>* old_access_regions,
+ const Array<BufferRegion>& infered_access_regions) {
+ auto fmutate = [this, &infered_access_regions](const BufferRegion&
buffer_region) {
+ if (buffer_region->buffer.same_as(old_buffer_)) {
+ ICHECK(infered_access_regions.size() == 1);
+ return infered_access_regions[0];
+ }
+ return buffer_region;
+ };
+ (*old_access_regions).MutateByApply(fmutate);
+ }
+
+ Stmt VisitStmt_(const BlockNode* op) final {
+ Block block = Downcast<Block>(StmtExprMutator::VisitStmt_(op));
Review comment:
Should the `BlockNode::alloc_buffers` be updated here? It's currently
updated down in line 160, but it feels like it would be more consistent to make
all the changes to the containing block in one place.
##########
File path: src/tir/schedule/primitive/layout_transformation.cc
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "../utils.h"
+
+namespace tvm {
+namespace tir {
+
+class TransformLayoutRewriter : private StmtExprMutator {
+ public:
+ /*!
+ * \brief Rewrite the access to the buffer after the transformation
+ * \param scope_stmt The parent statement that contains all accesses to the
target buffer
+ * \param old_buffer The target buffer before transformation
+ * \param new_buffer The new buffer after transformation
+ * \param index_map The transformation applied to the buffer
+ * \return The new AST rooting at the original parent scope and the map from
the old block to the
+ * new block
+ */
+ static std::pair<Stmt, Map<Block, Block>> Rewrite(const Stmt& scope_stmt,
+ const Buffer& old_buffer,
+ const Buffer& new_buffer,
+ const IndexMap& index_map)
{
+ TransformLayoutRewriter rewriter(old_buffer, new_buffer, index_map);
+ Stmt result = rewriter(scope_stmt);
+ return {result, rewriter.block_sref_reuse_};
+ }
+
+ private:
+ TransformLayoutRewriter(const Buffer& old_buffer, const Buffer& new_buffer,
+ const IndexMap& index_map)
+ : old_buffer_(old_buffer),
+ new_buffer_(new_buffer),
+ index_map_(index_map),
+ buffer_data_to_buffer_{{new_buffer->data, new_buffer}} {}
+
+ void RewriteBufferAccess(Buffer* buffer, Array<PrimExpr>* indices) {
+ *buffer = new_buffer_;
+ *indices = index_map_->MapIndices(*indices);
+ }
+
Review comment:
I'm guessing the answer is yes for the schedulable TIR, but I'd like to
verify: Is it safe to assume that all buffer allocations occur in `BlockNode`,
and that neither `AllocateNode` nor `AllocateConstNode` occur?
##########
File path: include/tvm/tir/schedule/schedule.h
##########
@@ -521,6 +522,21 @@ class ScheduleNode : public runtime::Object {
*/
virtual void Unannotate(const BlockRV& block_rv, const String& ann_key) = 0;
+ /******** Schedule: Layout transformation ********/
+ /*!
+ * \brief Apply a transformation represented by IndexMap to buffer
+ * \details The indices and the access region to the target buffer is
transformed by the given
+ * index_map. The index_map is used to infer the new shape of the buffer.
Buffer must be either
+ * a function parameter, or allocated in a block (it cannot be a buffer
subregion created via
+ * 'match_buffer').
Review comment:
Are the buffer subregions handled later on? That is, if a buffer is
transformed, are all `match_buffer` regions transformed to be consistent with
the transformation of the backing buffer?
##########
File path: tests/python/unittest/test_tir_schedule_transform_layout.py
##########
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=missing-function-docstring,missing-module-docstring
+import sys
+
+import pytest
+
+import tvm
+from tvm import tir
+from tvm.script import tir as T
+from tvm.tir.schedule.testing import verify_trace_roundtrip
+
+# fmt: off
+# pylint:
disable=no-member,invalid-name,unused-variable,line-too-long,redefined-outer-name,unexpected-keyword-arg,too-many-nested-blocks
+
+def packed_index_map_func(m, n):
+ return m // 16, n // 16, m % 16, n % 16
+
+
[email protected]_func
+def two_elementwise(a: T.handle, c: T.handle) -> None:
+ A = T.match_buffer(a, (128, 128), "float32")
Review comment:
Is there a benefit to using `T.match_buffer` rather than specifying the
parameter as `A: T.Buffer[(128,128), "float32"]`? If not, I'd lean toward
`T.Buffer` for readability.
##########
File path: src/tir/schedule/primitive/layout_transformation.cc
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+#include "../utils.h"
+
+namespace tvm {
+namespace tir {
+
+class TransformLayoutRewriter : private StmtExprMutator {
+ public:
+ /*!
+ * \brief Rewrite the access to the buffer after the transformation
+ * \param scope_stmt The parent statement that contains all accesses to the
target buffer
+ * \param old_buffer The target buffer before transformation
+ * \param new_buffer The new buffer after transformation
+ * \param index_map The transformation applied to the buffer
+ * \return The new AST rooting at the original parent scope and the map from
the old block to the
+ * new block
+ */
+ static std::pair<Stmt, Map<Block, Block>> Rewrite(const Stmt& scope_stmt,
+ const Buffer& old_buffer,
+ const Buffer& new_buffer,
+ const IndexMap& index_map)
{
+ TransformLayoutRewriter rewriter(old_buffer, new_buffer, index_map);
+ Stmt result = rewriter(scope_stmt);
+ return {result, rewriter.block_sref_reuse_};
+ }
+
+ private:
+ TransformLayoutRewriter(const Buffer& old_buffer, const Buffer& new_buffer,
+ const IndexMap& index_map)
+ : old_buffer_(old_buffer),
+ new_buffer_(new_buffer),
+ index_map_(index_map),
+ buffer_data_to_buffer_{{new_buffer->data, new_buffer}} {}
+
+ void RewriteBufferAccess(Buffer* buffer, Array<PrimExpr>* indices) {
+ *buffer = new_buffer_;
+ *indices = index_map_->MapIndices(*indices);
+ }
+
+ PrimExpr VisitExpr_(const BufferLoadNode* op) final {
+ BufferLoad buffer_load =
Downcast<BufferLoad>(StmtExprMutator::VisitExpr_(op));
+ if (buffer_load->buffer.same_as(old_buffer_)) {
+ auto* n = buffer_load.CopyOnWrite();
+ RewriteBufferAccess(&n->buffer, &n->indices);
+ }
+ return std::move(buffer_load);
+ }
+
+ Stmt VisitStmt_(const BufferStoreNode* op) final {
+ BufferStore buffer_store =
Downcast<BufferStore>(StmtExprMutator::VisitStmt_(op));
+ if (buffer_store->buffer.same_as(old_buffer_)) {
+ auto* n = buffer_store.CopyOnWrite();
+ RewriteBufferAccess(&n->buffer, &n->indices);
+ }
+ return std::move(buffer_store);
+ }
+
+ void RewriteAccessRegion(Array<BufferRegion>* old_access_regions,
+ const Array<BufferRegion>& infered_access_regions) {
+ auto fmutate = [this, &infered_access_regions](const BufferRegion&
buffer_region) {
+ if (buffer_region->buffer.same_as(old_buffer_)) {
+ ICHECK(infered_access_regions.size() == 1);
+ return infered_access_regions[0];
+ }
+ return buffer_region;
+ };
+ (*old_access_regions).MutateByApply(fmutate);
+ }
+
+ Stmt VisitStmt_(const BlockNode* op) final {
+ Block block = Downcast<Block>(StmtExprMutator::VisitStmt_(op));
+ auto infered_access_regions = GetBlockReadWriteRegion(block,
buffer_data_to_buffer_);
+ auto* n = block.CopyOnWrite();
+ RewriteAccessRegion(&n->reads, infered_access_regions[0]);
+ RewriteAccessRegion(&n->writes, infered_access_regions[1]);
+ block_sref_reuse_.Set(GetRef<Block>(op), block);
+ return std::move(block);
+ }
+
+ const Buffer& old_buffer_;
+ const Buffer& new_buffer_;
+ const IndexMap& index_map_;
+ Map<Var, Buffer> buffer_data_to_buffer_;
+ Map<Block, Block> block_sref_reuse_;
+};
+
+class BufferIsSubregionError : public ScheduleError {
+ public:
+ explicit BufferIsSubregionError(IRModule mod, Buffer buffer) : mod_(mod),
buffer_(buffer) {}
+
+ String FastErrorString() const final {
+ return "ScheduleError: The input buffer is defined in `match_buffer` of a
block, it is expected"
+ " to be a function parameter or allocated by a block";
+ }
+
+ String DetailRenderTemplate() const final {
+ std::ostringstream os;
+ os << "ScheduleError: The input buffer " << buffer_->name << " is defined
in `match_buffer` of "
+ << "a block, it is expected to be a function parameter or allocated by
a block.";
+ return os.str();
+ }
+
+ Array<ObjectRef> LocationsOfInterest() const final { return {}; }
+ IRModule mod() const final { return mod_; }
+
+ private:
+ IRModule mod_;
+ Buffer buffer_;
+};
+
+void TransformLayout(ScheduleState self, const StmtSRef& block_sref, int
buffer_index,
+ bool is_write_index, const IndexMap& index_map) {
+ const BlockNode* block_ptr = TVM_SREF_TO_BLOCK(block_ptr, block_sref);
+ Buffer old_buffer = GetNthAccessBuffer(self, GetRef<Block>(block_ptr),
buffer_index,
+ /*is_write=*/is_write_index);
+ Optional<StmtSRef> defining_site_sref;
+ bool is_alloc;
+ std::tie(defining_site_sref, is_alloc) = GetBufferDefiningSite(block_sref,
old_buffer);
+ if (defining_site_sref.defined() && !is_alloc) {
+ throw BufferIsSubregionError(self->mod, old_buffer);
+ }
+
+ StmtSRef scope_sref = defining_site_sref.defined()
+ ? defining_site_sref.value()
+ : GetScopeRoot(self, block_sref,
/*require_stage_pipeline=*/false);
+ const BlockNode* scope_block = TVM_SREF_TO_BLOCK(scope_block, scope_sref);
+
+ // Step 1: Infer the shape of the new buffer
+ ObjectPtr<BufferNode> new_buffer_node =
make_object<BufferNode>(*(old_buffer.get()));
+ new_buffer_node->shape = index_map->MapShape(old_buffer->shape);
+ Buffer new_buffer{new_buffer_node};
+
+ // Step 2: Rewrite access indices and regions of the buffer
+ Stmt new_stmt;
+ Map<Block, Block> block_sref_reuse;
+ std::tie(new_stmt, block_sref_reuse) = TransformLayoutRewriter::Rewrite(
+ GetRef<Block>(scope_block), old_buffer, new_buffer, index_map);
+ Block new_scope_block = Downcast<Block>(new_stmt);
+
Review comment:
Should this also rewrite the loop iterations surrounding the modified
buffer? I recall some discussion of modifying the loop iterations, but only if
the surrounding loop iterations were already in the same order as the
pre-transformation buffer's shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
