Michael ... many thanks for the excellent and informative follow up summary.
i too believe that these sorts of closing posts are very helpful to the newly Julian ... best, cdm On Thursday, January 15, 2015 at 2:07:18 PM UTC-8, Michael Wojnowicz wrote: > > > So I figured out the problem. Yes, as you both showed below, wrapping my > code snippet within a function seemed to get rid of memory allocation. But > my full code was still allocating memory, and I thought the problem came > from iterating OVER (rather than within) the function. I thought that the > results with these simple functions were misleading. > > In particular, I was running an optimization algorithm on a sparse > dataset of size 100,000 by 7.6 million, and, as I learned from the useful > command > > julia --track-allocation=user yourscript.jl. > my seemingly simple maximum/absolute value function was allocating, by the > time of convergence, about 20 GB of data ! > > As it turned out, my problem was indeed type stability, which people > describe as the #1 source of excess memory allocation. I had not > explicitly specified types in the form function(m::Float64,r::Float64); > rather I wrote my function in the form function(m,r). I assumed that > wasn't a problem, as discussed in the performance manual's section on "Type > Declarations," because my m values and r values were indeed always > Float64's. > > But alas I had initialized > m=0 > instead of > m=0.0 > > So I had initialized an integer, rather than a Float64. I had READ about > this mistake in the performance guide before! (It is discussed in "Avoid > changing the type of a variable"). But still, the "m=0" easily passed > inspection by my R- and MATLAB-trained eyes. And in the context of my > code, that 0 vs. 0.0 was the difference between allocating *20 GB* of > data and allocating only *104 bytes*! Amazing. > > Well, sorry to rehash the same old problems, but thanks for the help and > for helping me to discover this interesting example which really, painfully > emphasized an important difference between Julia and MATLAB/R for new > users. >
