Hi Mosè, Thanks for the wonderful feedback!
AutoGrad has some overhead for recording and differentiating primitive operators. However I think there is room for improvement - the current design has elements of the original Python package, anonymous functions, closures, etc. that are probably not very efficient. I am working on a more efficient design. Thanks for the derivative info as well! And here I was thinking: who is going to care about the missing derivative of the airy function :) I better get to work... best, deniz On Mon, Aug 29, 2016 at 2:13 AM Mosè Giordano <mose.giord...@gmail.com> wrote: > Hi Deniz, > > Announcing AutoGrad.jl <https://github.com/denizyuret/AutoGrad.jl>: an >> automatic differentiation package for Julia. It is a Julia port of the >> popular Python autograd <https://github.com/HIPS/autograd> package. It >> can differentiate regular Julia code that includes loops, conditionals, >> helper functions, closures etc. by keeping track of the primitive >> operations and using this execution trace to compute gradients. It uses >> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently >> handle functions with array inputs and scalar outputs. It can compute >> gradients of gradients to handle higher order derivatives. >> >> Large parts of the code are directly ported from the Python autograd >> <https://github.com/HIPS/autograd> package. I'd like to thank autograd >> author Dougal Maclaurin for his support. See (Baydin et al. 2015) >> <https://arxiv.org/abs/1502.05767> for a general review of automatic >> differentiation, autograd tutorial >> <https://github.com/HIPS/autograd/blob/master/docs/tutorial.md> for some >> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff >> <http://www.juliadiff.org/> has alternative differentiation tools for >> Julia. >> >> best, >> deniz >> >> > Very nice package! It's one of the most complete yet easy-to-use > automatic differentiation packages I've checked out. Some comments below. > > I expected (or hoped) that performance of the function returned by "grad" > was comparable to that of the actual derivative/gradient. However: > > julia> using BenchmarkTools, AutoGrad > > julia> COS = grad(sin) > gradfun (generic function with 1 method) > > julia> @benchmark cos(0.0) > BenchmarkTools.Trial: > samples: 10000 > evals/sample: 1000 > time tolerance: 5.00% > memory tolerance: 1.00% > memory estimate: 0.00 bytes > allocs estimate: 0 > minimum time: 4.00 ns (0.00% GC) > median time: 4.00 ns (0.00% GC) > mean time: 4.05 ns (0.00% GC) > maximum time: 15.00 ns (0.00% GC) > > julia> @benchmark COS(0.0) > BenchmarkTools.Trial: > samples: 10000 > evals/sample: 1 > time tolerance: 5.00% > memory tolerance: 1.00% > memory estimate: 3.61 kb > allocs estimate: 78 > minimum time: 9.14 μs (0.00% GC) > median time: 10.30 μs (0.00% GC) > mean time: 10.98 μs (3.46% GC) > maximum time: 3.88 ms (98.04% GC) > > > I found similar results for other simple functions. Is this to be > expected or is there room for improvements? > > I saw that the derivative dictionaries for most special functions are > incomplete. You can give a look at the derivatives in file src/math.jl > <https://github.com/giordano/Measurements.jl/blob/master/src/math.jl#L148> > of my Measurements.jl <https://github.com/giordano/Measurements.jl> > package (I could use an automatic differentiation library in my package, > but I've never found one that suited my needs). Look at the "result" calls > returned by the overloaded functions, the second argument is the derivative > or tuple of derivatives. > > I noticed that you marked the derivative of airyprime as wrong. I guess > that you've been deceived by the misleading docstring of airy: > https://github.com/JuliaLang/julia/issues/17032 The second derivatives > of airy functions can be computed exploiting the Airy equation > <https://en.wikipedia.org/wiki/Airy_function>. > > Cheers, > Mosè >
