Re: [julia-users] Re: AutoGrad port for Julia

[Deniz Yuret]

Hi Mosè,

Thanks for the wonderful feedback!

AutoGrad has some overhead for recording and differentiating primitive
operators.  However I think there is room for improvement - the current
design has elements of the original Python package, anonymous functions,
closures, etc. that are probably not very efficient.  I am working on a
more efficient design.

Thanks for the derivative info as well!  And here I was thinking: who is
going to care about the missing derivative of the airy function :)  I
better get to work...

best,
deniz


On Mon, Aug 29, 2016 at 2:13 AM Mosè Giordano 
wrote:

> Hi Deniz,
>
> Announcing AutoGrad.jl : an
>> automatic differentiation package for Julia. It is a Julia port of the
>> popular Python autograd  package. It
>> can differentiate regular Julia code that includes loops, conditionals,
>> helper functions, closures etc. by keeping track of the primitive
>> operations and using this execution trace to compute gradients. It uses
>> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently
>> handle functions with array inputs and scalar outputs. It can compute
>> gradients of gradients to handle higher order derivatives.
>>
>> Large parts of the code are directly ported from the Python autograd
>>  package. I'd like to thank autograd
>> author Dougal Maclaurin for his support. See (Baydin et al. 2015)
>>  for a general review of automatic
>> differentiation, autograd tutorial
>>  for some
>> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff
>>  has alternative differentiation tools for
>> Julia.
>>
>> best,
>> deniz
>>
>>
> Very nice package!  It's one of the most complete yet easy-to-use
> automatic differentiation packages I've checked out.  Some comments below.
>
> I expected (or hoped) that performance of the function returned by "grad"
> was comparable to that of the actual derivative/gradient.  However:
>
> julia> using BenchmarkTools, AutoGrad
>
> julia> COS = grad(sin)
> gradfun (generic function with 1 method)
>
> julia> @benchmark cos(0.0)
> BenchmarkTools.Trial:
>   samples:  1
>   evals/sample: 1000
>   time tolerance:   5.00%
>   memory tolerance: 1.00%
>   memory estimate:  0.00 bytes
>   allocs estimate:  0
>   minimum time: 4.00 ns (0.00% GC)
>   median time:  4.00 ns (0.00% GC)
>   mean time:4.05 ns (0.00% GC)
>   maximum time: 15.00 ns (0.00% GC)
>
> julia> @benchmark COS(0.0)
> BenchmarkTools.Trial:
>   samples:  1
>   evals/sample: 1
>   time tolerance:   5.00%
>   memory tolerance: 1.00%
>   memory estimate:  3.61 kb
>   allocs estimate:  78
>   minimum time: 9.14 μs (0.00% GC)
>   median time:  10.30 μs (0.00% GC)
>   mean time:10.98 μs (3.46% GC)
>   maximum time: 3.88 ms (98.04% GC)
>
>
> I found similar results for other simple functions.  Is this to be
> expected or is there room for improvements?
>
> I saw that the derivative dictionaries for most special functions are
> incomplete.  You can give a look at the derivatives in file src/math.jl
> 
> of my Measurements.jl 
> package (I could use an automatic differentiation library in my package,
> but I've never found one that suited my needs).  Look at the "result" calls
> returned by the overloaded functions, the second argument is the derivative
> or tuple of derivatives.
>
> I noticed that you marked the derivative of airyprime as wrong.  I guess
> that you've been deceived by the misleading docstring of airy:
> https://github.com/JuliaLang/julia/issues/17032  The second derivatives
> of airy functions can be computed exploiting the Airy equation
> .
>
> Cheers,
> Mosè
>

[julia-users] Re: AutoGrad port for Julia

[Mosè Giordano]

Hi Deniz,

Announcing AutoGrad.jl : an 
> automatic differentiation package for Julia. It is a Julia port of the 
> popular Python autograd  package. It 
> can differentiate regular Julia code that includes loops, conditionals, 
> helper functions, closures etc. by keeping track of the primitive 
> operations and using this execution trace to compute gradients. It uses 
> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently 
> handle functions with array inputs and scalar outputs. It can compute 
> gradients of gradients to handle higher order derivatives.
>
> Large parts of the code are directly ported from the Python autograd 
>  package. I'd like to thank autograd 
> author Dougal Maclaurin for his support. See (Baydin et al. 2015) 
>  for a general review of automatic 
> differentiation, autograd tutorial 
>  for some 
> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff 
>  has alternative differentiation tools for 
> Julia.
>
> best,
> deniz
>
>
Very nice package!  It's one of the most complete yet easy-to-use automatic 
differentiation packages I've checked out.  Some comments below.

I expected (or hoped) that performance of the function returned by "grad" 
was comparable to that of the actual derivative/gradient.  However:

julia> using BenchmarkTools, AutoGrad

julia> COS = grad(sin)
gradfun (generic function with 1 method)

julia> @benchmark cos(0.0)
BenchmarkTools.Trial: 
  samples:  1
  evals/sample: 1000
  time tolerance:   5.00%
  memory tolerance: 1.00%
  memory estimate:  0.00 bytes
  allocs estimate:  0
  minimum time: 4.00 ns (0.00% GC)
  median time:  4.00 ns (0.00% GC)
  mean time:4.05 ns (0.00% GC)
  maximum time: 15.00 ns (0.00% GC)

julia> @benchmark COS(0.0)
BenchmarkTools.Trial: 
  samples:  1
  evals/sample: 1
  time tolerance:   5.00%
  memory tolerance: 1.00%
  memory estimate:  3.61 kb
  allocs estimate:  78
  minimum time: 9.14 μs (0.00% GC)
  median time:  10.30 μs (0.00% GC)
  mean time:10.98 μs (3.46% GC)
  maximum time: 3.88 ms (98.04% GC)


I found similar results for other simple functions.  Is this to be expected 
or is there room for improvements?

I saw that the derivative dictionaries for most special functions are 
incomplete.  You can give a look at the derivatives in file src/math.jl 
 
of my Measurements.jl  package 
(I could use an automatic differentiation library in my package, but I've 
never found one that suited my needs).  Look at the "result" calls returned 
by the overloaded functions, the second argument is the derivative or tuple 
of derivatives.

I noticed that you marked the derivative of airyprime as wrong.  I guess 
that you've been deceived by the misleading docstring of airy: 
https://github.com/JuliaLang/julia/issues/17032  The second derivatives of 
airy functions can be computed exploiting the Airy equation 
.

Cheers,
Mosè

Re: [julia-users] Re: AutoGrad port for Julia

[Deniz Yuret]

I just discovered JuliaDiff , so I am not
entirely sure yet.  But here are my first impressions (JuliaDiff authors
please correct me if I am wrong):

* ForwardDiff : is not
practical (takes too much space) when you have large array inputs and
scalar outputs.

* ReverseDiffSource :
constructs gradient functions directly from the source code, which is fast
when it works, but as far as I can tell it doesn't work with non-trivial
code that has some combination of loops, conditionals, closures, helper
functions etc. This is ok for simple feed-forward models (if you are into
deep learning) but not sufficient for practical coding of bidirectional
RNNs etc.

* ReverseDiffOverload :
seems to follow the same strategy as AutoGrad.  I did not work out of the
box for me (Julia 0.4.6).

* ReverseDiffSparse : did
not have much documentation, but its intro says "automatic differentiation
for closed-form scalar algebraic expressions", so I suspect it doesn't do
arrays.

In summary AutoGrad  is intended
to work with models implemented in arbitrary Julia with large parameter
array inputs.  I am working on combining it with some GPU libraries for the
next version of the Knet  deep
learning framework.

best,
deniz


On Fri, Aug 26, 2016 at 3:59 PM Evan Fields 
wrote:

> This looks really interesting. Can you give the high level comparison of
> how this compares to the various JuliaDiff tools and when to use one over
> the other?
>

[julia-users] Re: AutoGrad port for Julia

[Evan Fields]

This looks really interesting. Can you give the high level comparison of 
how this compares to the various JuliaDiff tools and when to use one over 
the other?

[julia-users] Re: AutoGrad port for Julia

[michael . creel]

I've cloned it and tried it out with RC3. The mnist example runs fine. I'm 
looking forward to exploring it in more detail. Thanks!

On Friday, August 26, 2016 at 8:51:30 AM UTC+2, Deniz Yuret wrote:
>
> Announcing AutoGrad.jl : an 
> automatic differentiation package for Julia. It is a Julia port of the 
> popular Python autograd  package. It 
> can differentiate regular Julia code that includes loops, conditionals, 
> helper functions, closures etc. by keeping track of the primitive 
> operations and using this execution trace to compute gradients. It uses 
> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently 
> handle functions with array inputs and scalar outputs. It can compute 
> gradients of gradients to handle higher order derivatives.
>
> Large parts of the code are directly ported from the Python autograd 
>  package. I'd like to thank autograd 
> author Dougal Maclaurin for his support. See (Baydin et al. 2015) 
>  for a general review of automatic 
> differentiation, autograd tutorial 
>  for some 
> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff 
>  has alternative differentiation tools for 
> Julia.
>
> best,
> deniz
>
>

Re: [julia-users] Re: AutoGrad port for Julia

[Henri Girard]

your welcome :)
Le 26/08/2016 à 10:49, Deniz Yuret a écrit :

Sorry about that.  Here is the right link:

https://github.com/denizyuret/AutoGrad.jl


On Fri, Aug 26, 2016 at 11:38 AM Henri Girard > wrote:


Your link autograd.jl seems dead ?


Le vendredi 26 août 2016 08:51:30 UTC+2, Deniz Yuret a écrit :

Announcing AutoGrad. jl
: an automatic
differentiation package for Julia. It is a Julia port of the
popular Python autograd
 package. It can
differentiate regular Julia code that includes loops,
conditionals, helper functions, closures etc. by keeping track
of the primitive operations and using this execution trace to
compute gradients. It uses reverse mode differentiation
(a.k.a. backpropagation) so it can efficiently handle
functions with array inputs and scalar outputs. It can compute
gradients of gradients to handle higher order derivatives.

Large parts of the code are directly ported from the Python
autograd  package. I'd like
to thank autograd author Dougal Maclaurin for his support. See
(Baydin et al. 2015)  for a
general review of automatic differentiation, autograd tutorial
 for
some Python examples, and Dougal's PhD thesis for design
principles. JuliaDiff  has
alternative differentiation tools for Julia.

best,
deniz

Re: [julia-users] Re: AutoGrad port for Julia

[Deniz Yuret]

Sorry about that.  Here is the right link:

https://github.com/denizyuret/AutoGrad.jl


On Fri, Aug 26, 2016 at 11:38 AM Henri Girard 
wrote:

> Your link autograd.jl seems dead ?
>
>
> Le vendredi 26 août 2016 08:51:30 UTC+2, Deniz Yuret a écrit :
>>
>> Announcing AutoGrad.jl : an
>> automatic differentiation package for Julia. It is a Julia port of the
>> popular Python autograd  package. It
>> can differentiate regular Julia code that includes loops, conditionals,
>> helper functions, closures etc. by keeping track of the primitive
>> operations and using this execution trace to compute gradients. It uses
>> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently
>> handle functions with array inputs and scalar outputs. It can compute
>> gradients of gradients to handle higher order derivatives.
>>
>> Large parts of the code are directly ported from the Python autograd
>>  package. I'd like to thank autograd
>> author Dougal Maclaurin for his support. See (Baydin et al. 2015)
>>  for a general review of automatic
>> differentiation, autograd tutorial
>>  for some
>> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff
>>  has alternative differentiation tools for
>> Julia.
>>
>> best,
>> deniz
>>
>>

[julia-users] Re: AutoGrad port for Julia

[Henri Girard]

Your link autograd.jl seems dead ?

Le vendredi 26 août 2016 08:51:30 UTC+2, Deniz Yuret a écrit :
>
> Announcing AutoGrad.jl : an 
> automatic differentiation package for Julia. It is a Julia port of the 
> popular Python autograd  package. It 
> can differentiate regular Julia code that includes loops, conditionals, 
> helper functions, closures etc. by keeping track of the primitive 
> operations and using this execution trace to compute gradients. It uses 
> reverse mode differentiation (a.k.a. backpropagation) so it can efficiently 
> handle functions with array inputs and scalar outputs. It can compute 
> gradients of gradients to handle higher order derivatives.
>
> Large parts of the code are directly ported from the Python autograd 
>  package. I'd like to thank autograd 
> author Dougal Maclaurin for his support. See (Baydin et al. 2015) 
>  for a general review of automatic 
> differentiation, autograd tutorial 
>  for some 
> Python examples, and Dougal's PhD thesis for design principles. JuliaDiff 
>  has alternative differentiation tools for 
> Julia.
>
> best,
> deniz
>
>