Sorry, yeah, I screwed up sending dsoftmax (somehow I sent an older version of the function).
Anyway, thanks. Your method seems to be fast. I'm going to test it against sm D. 1 Thanks. -------------------------------------------- On Tue, 2/28/17, 'Mike Day' via Programming <[email protected]> wrote: Subject: Re: [Jprogramming] Fast derivative of Softmax function To: [email protected] Date: Tuesday, February 28, 2017, 1:05 AM It's not clear what you had in mind for the dyadic function "softmax" . Anyway, is this what you require? I =: =/~@:i.@# NB. Identity matrix (Kronecker delta) pdsm =: (*"1 (-~I))@:sm NB. Partial derivative (matrix) of sm pdsm a 0.186192 _0.0676431 _0.0467234 _0.0718259 _0.0676431 0.19866 _0.0516374 _0.0793799 _0.0467234 _0.0516374 0.153191 _0.0548305 _0.0718259 _0.0793799 _0.0548305 0.206036 This returns the whole matrix, but it's easy enough to save the output, and extract the indexed elements you need. Or have I misunderstood the specification? Mike On 27/02/2017 08:09, 'Jon Hough' via Programming wrote: > Given an array, we can calculate the softmax function > https://en.wikipedia.org/wiki/Softmax_function > > a =: 0.5 0.6 0.23 0.66 > sm=:(] % +/ )@:^ NB. softmax > > sm a > 0.247399 0.273418 0.188859 0.290325 > > The (partial) derivative of softmax is a little more complicated: > > If the array is of length N, we need an NxN matrix of partial derivatives where (in pseudo code) > > derivatives[i,j] = sm (array[i] ) *( 1 - sm(array[j]) if i == j > or > derivatives[i,j] = -1 * sm (array[i] ) * ( sm(array[j]) if i != j > > ( see here for the reasoning: http://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/ ) > > My implementation of the partial derivatives is this: > > > NB. x value is index, y value is the whole array > dsoftmax=: 4 : 0 > idx=. x > vals=. y > smx=. idx softmax vals > rx=. '' > for_j. i.#vals do. > if. j = idx do. rx=. rx , smx * (1 - smx) > elseif. 1 do. rx=. rx ,(j softmax vals)* (0 - smx) end. > end. > rx > ) > > > Then, for example using above array a, > > (i.# a) dsoftmax"0 _ a > > gives the values, in a 4x4 matrix. > > This is quite slow. I have tried to do this without iterating and branching, but cannot figure out a way to do it. > Any help appreciated. > Thanks, > > Jon > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
