@Jeff, do we account for the amount of memory handed to the gc in this use case? It seems the gc might not realize that a collection would be beneficial.
If you call c_free after pointer_to_array(own=true), your program will (eventually / randomly) segfault. Have you explicitly called `gc()`, to confirm that the memory is not released? On Tue, May 6, 2014 at 4:29 PM, Gustavo Camilo <[email protected]> wrote: > I am passing true to it (see previous post), but the leak still happens. I'm > testing it currently and I think c_free(pointer) after pointer_to_array > fixes the problem. Are you sure we don't have to free the pointer? > > > On Tuesday, 6 May 2014 16:25:10 UTC-4, James Porter wrote: >> >> By default pointer_to_array will not cause the Julia runtime take >> ownership of the array, but you can optionally pass `own=true` to cause this >> to happen, and then the memory will be garbage collected when it is no >> longer referenced. You can also find this information at the REPL by typing >> `help(ponter_to_array)` >> >> On Tuesday, May 6, 2014 3:18:19 PM UTC-5, Gustavo Camilo wrote: >>> >>> Am I supposed to run c_free(pointer) after using pointer_to_array (and >>> before returning my c wrapper function)? >>> >>> On Tuesday, 6 May 2014 15:13:08 UTC-4, Gustavo Camilo wrote: >>>> >>>> But if the C code returns a pointer to the malloced data, and I >>>> pointer_to_array(pointer,size,true), it should take ownership of it, and >>>> properly garbage it? >>>> >>>> On Tuesday, 6 May 2014 15:10:04 UTC-4, James Porter wrote: >>>>> >>>>> The Julia garbage collector does not know about or manage memory you >>>>> allocate in external C calls, if you free that memory yourself in your C >>>>> code you should see your memory usage improve dramatically. >>>>> >>>>> On Tuesday, May 6, 2014 1:33:30 PM UTC-5, Gustavo Camilo wrote: >>>>>> >>>>>> Hi all, I have a related question so I thought I would bump it up: >>>>>> >>>>>> I call a C function to do a lot of work using pmap in my code. Inside >>>>>> my C function I malloc and return arrays of floats. These then are used >>>>>> to >>>>>> perform some other computations. >>>>>> >>>>>> However I've put this inside an optimizer so now these functions are >>>>>> called over and over again. However my memory usage blows up, as if after >>>>>> ending the >>>>>> function call the data that was malloc'ed is not released. >>>>>> >>>>>> What I am wondering is if every time the optimizer calls this, since >>>>>> several variables were malloc'ed in C, >>>>>> does the Julia garbage collector get rid of them, or should I pass >>>>>> empty matrices the the C function. >>>>>> >>>>>> Thanks for your help, first is a sketch of each C call: >>>>>> >>>>>> for i in 1:length(parameterSpace) >>>>>> results[i] = pmap(cFunction,parameters[i]) >>>>>> end >>>>>> >>>>>> (inside of every cFunction call there was a malloc) >>>>>> >>>>>> and then linear indexing to recuperate the results >>>>>> matrix = Array(Float64,size1,size2,...) >>>>>> for s1 in 1:size1, s2 in 1:size2 >>>>>> matrix[s1,s2] = results[s1 + s2] >>>>>> end >>>>>> >>>>>> >>>>>> >>>>>> Here is the objective function of the optimization >>>>>> >>>>>> function computeModel(parameters::Vector) >>>>>> theta1 = parameters[1] >>>>>> theta2 = parameters[2] >>>>>> betaG = parameters[3] >>>>>> chi = parameters[4] >>>>>> alpha1 = parameters[5] >>>>>> alpha2 = parameters[6] >>>>>> rho1 = parameters[7] >>>>>> rho2 = parameters[8] >>>>>> sigma1 = parameters[9] >>>>>> sigma2 = parameters[10] >>>>>> k0 = parameters[11] >>>>>> g0 = parameters[12] >>>>>> >>>>>> sizeV = 20 >>>>>> nsim = 1000 >>>>>> age = 40 >>>>>> >>>>>> println("-----------------------------------------------------") >>>>>> println("New Iteration. Parameters: ",parameters) >>>>>> println("Compute Priv") >>>>>> privBase = >>>>>> ModelBase(0.96,betaG,0.05,alpha1,chi,theta1,sigma1,rho1,g0,sizeV,sizeV,sizeV,sizeV,sizeV,k0) >>>>>> privModel = genModelPrivate(privBase) >>>>>> newBounds,guess,capitals = findBoundsPrivateInC(privModel) >>>>>> vf,pk,pq1,pq2,pq3 = >>>>>> runVFIPrivInC(newBounds,guess,tol=.003,capitals=capitals) >>>>>> sk,sq1,sq2,sq3,sd,sw,sz,sq = >>>>>> simulatePriv(newBounds,pk,pq1,pq2,pq3,capitals,nsim,age) >>>>>> momentsPriv = computeMoments(sk,sw,sd,sz,sq,newBounds) >>>>>> >>>>>> >>>>>> println("Compute SOE") >>>>>> soeBase = >>>>>> ModelBase(0.96,betaG,0.05,alpha2,chi,theta2,sigma2,rho2,g0,sizeV,sizeV,sizeV,sizeV,sizeV,k0) >>>>>> soeModel = genModelSOE(soeBase) >>>>>> >>>>>> newBoundsSoe,guessSoe,capitalsSoe = findBoundsSOEInC(soeModel) >>>>>> newValueFPareto,pK,pQ1,pQ2,pQ3 = >>>>>> runVFISOEInC(newBoundsSoe,guess=guessSoe,tol=.003,capitals=capitalsSoe) >>>>>> nVF,promiseMap = >>>>>> findPromiseSpace(newValueFPareto,newBoundsSoe,pK,pQ1,pQ2,pQ3) >>>>>> sk,sq1,sq2,sq3,sd,sw,sz,sq = >>>>>> simulateSOE(newBoundsSoe,pK,pQ1,pQ2,pQ3,capitalsSoe,promiseMap,nsim,age) >>>>>> momentsSoe = computeMoments(sk,sw,sd,sz,sq,newBoundsSoe) >>>>>> >>>>>> popMoments = >>>>>> [0.0830,0.1215,0.0599,0.1243,0.0039,0.016,0.3432,0.6291,0.0834,0.1542,0.0651,0.2109,0.0079,0.02,0.6705,1.2212] >>>>>> >>>>>> weight = eye(length(popMoments)) >>>>>> g = float64(vcat([momentsPriv[i] for >>>>>> i=1:length(momentsPriv)],[momentsSoe[i] for i=1:length(momentsSoe)])) - >>>>>> popMoments >>>>>> >>>>>> value = (g'*weight*g)[1] >>>>>> println("f(x) = ", value) >>>>>> >>>>>> return value >>>>>> end >>>>>> >>>>>> >>>>>> On Wednesday, 16 April 2014 19:21:29 UTC-4, Gustavo Camilo wrote: >>>>>>> >>>>>>> Hi all, I'm currently using pmap to distribute to a bunch of cores >>>>>>> some work to do. I was just wondering how you guys are structuring this. >>>>>>> >>>>>>> Basically I have a function written in C that I need to pass a bunch >>>>>>> of parameters to so that it can do its work, so I first create a tuple >>>>>>> with >>>>>>> all the parameters needed >>>>>>> for one worker to find results, for every combination of parameters: >>>>>>> >>>>>>> (I'm solving an infinite dimensional programming problem using >>>>>>> bellman equations and brute force to maximize, the C code searches >>>>>>> through >>>>>>> the control space >>>>>>> and finds the best choices to maximize the objective) >>>>>>> >>>>>>> input = >>>>>>> [(w,z,mu,model,currentValue[:,:,nextPromise[mu]],squeeze(squeeze(capitals[w,z,:,:,:,:],2),1)) >>>>>>> for w in 1:nW, z in 1:nZ, mu in 1:nMu]; >>>>>>> >>>>>>> And then I simply pass it with pmap to the wrapper of my C function: >>>>>>> >>>>>>> results = pmap(findPolicyC,input) >>>>>>> >>>>>>> And use linear indexing to recuperate the answers: >>>>>>> >>>>>>> anArray = results[w + nW*((z-1) + nZ*(mu-1))] >>>>>>> >>>>>>> >>>>>>> Is this how you guys would structure this parallel computation? I've >>>>>>> attached the code for any that are interested. You need GSL to run the C >>>>>>> part: >>>>>>> Eg. compile with, where the directories point to GSL: gcc >>>>>>> -I/home/gcam/china/gsl -L/home/gcam/china/gsl/.libs -shared -fPIC >>>>>>> findPolicySOE.c -lgsl -lm -o findPSoe.so >>>>>>> >>>>>>> >>>>>>> Thanks for the input!
