altanh opened a new pull request #7677: URL: https://github.com/apache/tvm/pull/7677
Since there is negligible code sharing between the first-order and higher-order AD in Relay, I've factored out the first-order AD pass to a separate file. Additionally, I've made it a proper IRModule pass, and fixed a few spots where adding tuples wasn't being lifted correctly. I tried my best to add some diagnostics. To complement AD, I've also added a pass called `ConcretizeLike` that transforms the `*_like` operators generated during AD to the explicit-shape equivalent (e.g. `zeros_like(x, y) -> zeros(x, y.shape)`), which removes some forward dependencies and could enable more opportunity for operator fusion. Note that this is the same thing as `LikeZapp` from @t-vi's [blogpost](https://lernapparat.de/transformers-pytorch-tvm/) with perhaps a bit more error checking. cc @t-vi @MarisaKirisame @tqchen @jroesch ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
