MarisaKirisame commented on issue #4432: [Relay][Pass] Fix lambda lift pass for recursive call URL: https://github.com/apache/incubator-tvm/pull/4432#issuecomment-559238842 @junrushao1994 No need to say that! DL compiler require many expertise and we are all ignorant in different things. I should answer the general problem before the specific problem. But before we start, in general, closure is the "killer" of ad. Some ad work do not support closure, like tangent or tensorflow-for-swift. Some work support it, but take lots of effort - Lambda The Ultimate BackPropagator and differentiate through a curry (by author of tf for swift) come to mind. Additionally, all work need some additional bookkeeping for captured value inside the closure, in order to backprop correctly to them. (If we do not use reference, the caller of a lambda must update the reference itself.) Some people use reflection, and some people use some dynamic typing and cast. Reference is a very lightweight bookkeeping technique while making stuff good for other optimization (Reflection will be a much bigger problem for opt/codegen then ref). ### How to do autodiff of lambda with minimal amount of reference? The method of turning lambdas into datastructure is well known and it is called defunctionalization. Basically, it try to make a new datatype, Func, where every single constructor is a "lambda" in the AST, and the constructor take in the closure number. For example, suppose the AST contain the following lambda, all taking two int and returning int (fun _ _ -> a), (fun a b -> a * a + b * b) (fun a b -> a + b + c) (fun a b -> f (f a b) (f b a)) the following adt will be declared: type FUN = | CONST of int | AddSquare | AddThree of int | Point of FUN. There is some issue regarding making it type safe, but I imagine we can get by by using some cast here and there. we can then turn some adt containing reference into reference of pure adt. After that we can then do store passing style to that single reference. But I do not think that we can in general remove it (I had tried like 10 times). Off a side note, why are you so eager to remove reference? I need to write paper and I think this is a good paper story. I will talk about lambda lifting below, but I think we should not mix up lambda lifting with defunctionalization.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
