The Julia garbage collector does not know about or manage memory you
allocate in external C calls, if you free that memory yourself in your C
code you should see your memory usage improve dramatically.
On Tuesday, May 6, 2014 1:33:30 PM UTC-5, Gustavo Camilo wrote:
>
> Hi all, I have a related question so I thought I would bump it up:
>
> I call a C function to do a lot of work using pmap in my code. Inside my C
> function I malloc and return arrays of floats. These then are used to
> perform some other computations.
>
> However I've put this inside an optimizer so now these functions are
> called over and over again. However my memory usage blows up, as if after
> ending the
> function call the data that was malloc'ed is not released.
>
> *What I am wondering is if every time the optimizer calls this, since
> several variables were malloc'ed in C,*
> *does the Julia garbage collector get rid of them, or should I pass empty
> matrices the the C function.*
>
> Thanks for your help, first is a sketch of each C call:
>
> for i in 1:length(parameterSpace)
> results[i] = pmap(cFunction,parameters[i])
> end
>
> (inside of every cFunction call there was a malloc)
>
> and then linear indexing to recuperate the results
> matrix = Array(Float64,size1,size2,...)
> for s1 in 1:size1, s2 in 1:size2
> matrix[s1,s2] = results[s1 + s2]
> end
>
>
>
> Here is the objective function of the optimization
>
> function computeModel(parameters::Vector)
> theta1 = parameters[1]
> theta2 = parameters[2]
> betaG = parameters[3]
> chi = parameters[4]
> alpha1 = parameters[5]
> alpha2 = parameters[6]
> rho1 = parameters[7]
> rho2 = parameters[8]
> sigma1 = parameters[9]
> sigma2 = parameters[10]
> k0 = parameters[11]
> g0 = parameters[12]
>
> sizeV = 20
> nsim = 1000
> age = 40
>
> println("-----------------------------------------------------")
> println("New Iteration. Parameters: ",parameters)
> println("Compute Priv")
> privBase =
> ModelBase(0.96,betaG,0.05,alpha1,chi,theta1,sigma1,rho1,g0,sizeV,sizeV,sizeV,sizeV,sizeV,k0)
> privModel = genModelPrivate(privBase)
> newBounds,guess,capitals = findBoundsPrivateInC(privModel)
> vf,pk,pq1,pq2,pq3 =
> runVFIPrivInC(newBounds,guess,tol=.003,capitals=capitals)
> sk,sq1,sq2,sq3,sd,sw,sz,sq =
> simulatePriv(newBounds,pk,pq1,pq2,pq3,capitals,nsim,age)
> momentsPriv = computeMoments(sk,sw,sd,sz,sq,newBounds)
>
>
> println("Compute SOE")
> soeBase =
> ModelBase(0.96,betaG,0.05,alpha2,chi,theta2,sigma2,rho2,g0,sizeV,sizeV,sizeV,sizeV,sizeV,k0)
> soeModel = genModelSOE(soeBase)
>
> newBoundsSoe,guessSoe,capitalsSoe = findBoundsSOEInC(soeModel)
> newValueFPareto,pK,pQ1,pQ2,pQ3 =
> runVFISOEInC(newBoundsSoe,guess=guessSoe,tol=.003,capitals=capitalsSoe)
> nVF,promiseMap =
> findPromiseSpace(newValueFPareto,newBoundsSoe,pK,pQ1,pQ2,pQ3)
> sk,sq1,sq2,sq3,sd,sw,sz,sq =
> simulateSOE(newBoundsSoe,pK,pQ1,pQ2,pQ3,capitalsSoe,promiseMap,nsim,age)
> momentsSoe = computeMoments(sk,sw,sd,sz,sq,newBoundsSoe)
>
> popMoments =
> [0.0830,0.1215,0.0599,0.1243,0.0039,0.016,0.3432,0.6291,0.0834,0.1542,0.0651,0.2109,0.0079,0.02,0.6705,1.2212]
>
> weight = eye(length(popMoments))
> g = float64(vcat([momentsPriv[i] for
> i=1:length(momentsPriv)],[momentsSoe[i] for i=1:length(momentsSoe)])) -
> popMoments
>
> value = (g'*weight*g)[1]
> println("f(x) = ", value)
>
> return value
> end
>
>
> On Wednesday, 16 April 2014 19:21:29 UTC-4, Gustavo Camilo wrote:
>>
>> Hi all, I'm currently using pmap to distribute to a bunch of cores some
>> work to do. I was just wondering how you guys are structuring this.
>>
>> Basically I have a function written in C that I need to pass a bunch of
>> parameters to so that it can do its work, so I first create a tuple with
>> all the parameters needed
>> for one worker to find results, for every combination of parameters:
>>
>> (I'm solving an infinite dimensional programming problem using bellman
>> equations and brute force to maximize, the C code searches through the
>> control space
>> and finds the best choices to maximize the objective)
>>
>> input =
>> [(w,z,mu,model,currentValue[:,:,nextPromise[mu]],squeeze(squeeze(capitals[w,z,:,:,:,:],2),1))
>>
>> for w in 1:nW, z in 1:nZ, mu in 1:nMu];
>>
>> And then I simply pass it with pmap to the wrapper of my C function:
>>
>> results = pmap(findPolicyC,input)
>>
>> And use linear indexing to recuperate the answers:
>>
>> anArray = results[w + nW*((z-1) + nZ*(mu-1))]
>>
>>
>> Is this how you guys would structure this parallel computation? I've
>> attached the code for any that are interested. You need GSL to run the C
>> part:
>> Eg. compile with, where the directories point to GSL: gcc
>> -I/home/gcam/china/gsl -L/home/gcam/china/gsl/.libs -shared -fPIC
>> findPolicySOE.c -lgsl -lm -o findPSoe.so
>>
>>
>> Thanks for the input!
>>
>
