I installed valgrind 3.11 SVN with homebrew and tried to run the code but I
am not familiar with the generated output.
I added a valgrind-julia.supp and used the command parameters explained here
<https://github.com/JuliaLang/julia/blob/master/doc/devdocs/valgrind.rst>
I executed:
valgrind --smc-check=all-non-file --suppressions=valgrind-julia.supp
/Applications/Julia-dev.app/Contents/Resources/julia/bin/julia
simulation.jl a.out > log.txt 2>&1
I attach the generated log.txt file.
Bartolomeo
Il giorno giovedì 12 marzo 2015 03:10:10 UTC, Tim Holy ha scritto:
>
> On Wednesday, March 11, 2015 05:33:10 PM Bartolomeo Stellato wrote:
> > I also tried with Julia 0.4.0-dev+3752 and I encounter the same problem.
>
> Hm. If you're sure there's a leak, this should be investigated. Any chance
> you
> can try valgrind?
>
> --Tim
>
> >
> > Il giorno mercoledì 11 marzo 2015 22:51:18 UTC, Tony Kelman ha scritto:
> > > The majority of the memory allocation is almost definitely coming from
> the
> > > problem setup here. You're using a dense block-triangular fomulation
> of
> > > MPC, eliminating states and only solving for inputs with inequality
> > > constraints. Since you're converting your problem data to sparse
> > > initially,
> > > you're doing a lot of extra allocation, integer arithmetic, and
> consuming
> > > more memory to represent a large dense matrix in sparse format.
> > > Reformulate
> > > your problem to include both states and inputs as unknowns, and
> enforce
> > > the
> > > dynamics as equality constraints. This will result in a block-banded
> > > problem structure and maintain sparsity much better. The matrices
> within
> > > the blocks are not sparse here since you're doing an exact
> discretization
> > > with expm, but a banded problem will scale much better to longer
> horizons
> > > than a triangular one.
> > >
> > > You also should be able to reuse the problem data, with the exception
> of
> > > bounds or maybe vector coefficients, between different MPC iterations.
> > >
> > >
> > >
> > > On Wednesday, March 11, 2015 at 12:14:03 PM UTC-7, Bartolomeo Stellato
> > >
> > > wrote:
> > >> Thank you for the quick replies and for the suggestions!
> > >>
> > >> I checked which lines give more allocation with
> --track-allocation=user
> > >> and the amount of memory I posted is from the OSX Activity monitor.
> > >> Even if it is not all necessarily used, if it grows too much the
> > >> operating system is forced to kill Julia.
> > >>
> > >> I slightly edited the code in order to *simulate the closed loop 6
> times*
> > >> (for different parameters of N and lambdau). I attach the files. The*
> > >> allocated memory *with the OSX Activity monitor is *2gb now.*
> > >> If I run the code twice with a clear_malloc_data() in between to save
> > >> --track-allocation=user information I get something around 3.77gb!
> > >>
> > >> Are there maybe problems with my code for which the allocated memory
> > >> increases? I can't understand why by simply running the same function
> 6
> > >> times, the memory increases so much. Unfortunately I need to do it
> > >> hundreds
> > >> of times in this way it is impossible.
> > >>
> > >> Do you think that using the push! function together with reducing the
> > >> vector computations could significantly reduce this big amount of
> > >> allocated
> > >> memory?
> > >>
> > >>
> > >> Bartolomeo
> > >>
> > >> Il giorno mercoledì 11 marzo 2015 17:07:23 UTC, Tim Holy ha scritto:
> > >>> --track-allocation doesn't report the _net_ memory allocated, it
> reports
> > >>> the
> > >>> _gross_ memory allocation. In other words, allocate/free adds to the
> > >>> tally,
> > >>> even if all memory is eventually freed.
> > >>>
> > >>> If you're still concerned about memory allocation and its likely
> impact
> > >>> on
> > >>> performance: there are some things you can do. From glancing at your
> > >>> code very
> > >>> briefly, a couple of comments:
> > >>> - My crystal ball tells me you will soon come to adore the push!
> > >>> function :-)
> > >>> - If you wish (and it's your choice), you can reduce allocations by
> > >>> doing more
> > >>> operations with scalars. For example, in computeReferenceCurrents,
> > >>> instead of
> > >>> computing tpu and iref arrays outside the loop, consider performing
> the
> > >>> equivalent operations on scalar values inside the loop.
> > >>>
> > >>> Best,
> > >>> --Tim
> > >>>
> > >>> On Wednesday, March 11, 2015 07:41:19 AM Bartolomeo Stellato wrote:
> > >>> > Hi all,
> > >>> >
> > >>> > I recently started using Julia for my *Closed loop MPC
> simulations.* I
> > >>>
> > >>> fond
> > >>>
> > >>> > very interesting the fact that I was able to do almost everything
> I
> > >>>
> > >>> was
> > >>>
> > >>> > doing in MATLAB with Julia. Unfortunately, when I started working
> on
> > >>>
> > >>> more
> > >>>
> > >>> > complex simulations I notice a *memory allocation problem*.
> > >>> >
> > >>> > I am using OSX Yosemite and Julia 0.3.6. I attached a MWE that can
> be
> > >>> > executed with include("simulation.jl")
> > >>> >
> > >>> > The code executes a single simulation of the closed loop system
> with a
> > >>>
> > >>> *MPC
> > >>>
> > >>> > controller* solving an optimization problem at each time step via
> > >>>
> > >>> *Gurobi
> > >>>
> > >>> > interface*. At the end of the simulation I am interested in only
> *two
> > >>> > performance indices* (float numbers).
> > >>> > The simulation, however, takes more than 600mb of memory and, even
> if
> > >>>
> > >>> most
> > >>>
> > >>> > of the declared variables local to different functions, I can't
> get
> > >>>
> > >>> rid of
> > >>>
> > >>> > them afterwards with the garbage collector: gc()
> > >>> >
> > >>> > I analyzed the memory allocation with julia
> --track-allocation=user
> > >>>
> > >>> and I
> > >>>
> > >>> > included the generated .mem files. Probably my code is not
> optimized,
> > >>>
> > >>> but I
> > >>>
> > >>> > can't understand *why all that memory doesn't get deallocated
> after
> > >>>
> > >>> the
> > >>>
> > >>> > simulation*.
> > >>> >
> > >>> > Is there anyone who could give me any explanation or suggestion to
> > >>>
> > >>> solve
> > >>>
> > >>> > that problem? I need to perform several of these simulations and
> it is
> > >>> > impossible for me to allocate for each one more than 600mb.
> > >>> >
> > >>> >
> > >>> > Thank you!
> > >>> >
> > >>> > Bartolomeo
>
>==9270== Memcheck, a memory error detector
==9270== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==9270== Using Valgrind-3.11.0.SVN and LibVEX; rerun with -h for copyright info
==9270== Command:
/Applications/Julia-dev.app/Contents/Resources/julia/bin/julia simulation.jl
a.out
==9270==
==9270== Warning: set address range perms: large range [0x239a6c000,
0x43a780000) (defined)
UNKNOWN thread message [id 3613, to mach_thread_self(), reply 0x30f]
==9270== Conditional jump or move depends on uninitialised value(s)
==9270== at 0x100165647:
llvm::X86TargetLowering::LowerCall(llvm::TargetLowering::CallLoweringInfo&,
llvm::SmallVectorImpl<llvm::SDValue>&) const (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1002EFF26:
llvm::TargetLowering::LowerCallTo(llvm::TargetLowering::CallLoweringInfo&)
const (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1002DD145:
llvm::SelectionDAGBuilder::LowerCallTo(llvm::ImmutableCallSite, llvm::SDValue,
bool, llvm::MachineBasicBlock*) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1002CF6DF: llvm::SelectionDAGBuilder::visitCall(llvm::CallInst
const&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1002C9989: llvm::SelectionDAGBuilder::visit(unsigned int,
llvm::User const&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1002C8EE1: llvm::SelectionDAGBuilder::visit(llvm::Instruction
const&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x100302B87:
llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::Instruction
const>, llvm::ilist_iterator<llvm::Instruction const>, bool&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x100302AB5:
llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1003017C3:
llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x100512D02:
llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1009598FC: llvm::FPPassManager::runOnFunction(llvm::Function&)
(in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x10095936E: llvm::FunctionPassManagerImpl::run(llvm::Function&)
(in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270==
==9270== Conditional jump or move depends on uninitialised value(s)
==9270== at 0x100556768: (anonymous
namespace)::PeepholeOptimizer::runOnMachineFunction(llvm::MachineFunction&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x100512D02:
llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x1009598FC: llvm::FPPassManager::runOnFunction(llvm::Function&)
(in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x10095936E: llvm::FunctionPassManagerImpl::run(llvm::Function&)
(in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x100959275: llvm::FunctionPassManager::run(llvm::Function&) (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x10048D1CD: llvm::JIT::getPointerToFunction(llvm::Function*)
(in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib)
==9270== by 0x10002D4E1: jl_generate_fptr (codegen.cpp:712)
==9270== by 0x1000287D5: jl_trampoline_compile_function (builtins.c:920)
==9270== by 0x100028863: jl_trampoline (builtins.c:931)
==9270== by 0x100020B98: jl_apply_generic (julia.h:1089)
==9270== by 0x1070F0CB7: julia_print_to_string_25956 (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib)
==9270== by 0x107298C43: julia_init_stdio_41084 (in
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib)
==9270==
Warning: error initializing module Base:
OverflowError()
Warning: error initializing module LinAlg:
OverflowError()
fatal: error thrown and no exception handler available.
OverflowError()
rec_backtrace at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/task.c:638
jl_throw_with_superfluous_argument at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/libjulia.dylib
(unknown line)
typeinf_uncached at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jlcall_typeinf_uncached_25870 at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jl_apply at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/./julia.h:1089
typeinf at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jlcall_typeinf_25885 at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jl_apply at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/./julia.h:1089
typeinf_ext at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jl_apply at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/./julia.h:1089
jl_apply at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/./julia.h:496
jl_mt_assoc_by_type at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/gf.c:1069
jl_apply_generic at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/gf.c:1694
_start at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jlcall__start_40683 at
/Applications/Julia-dev.app/Contents/Resources/julia/lib/julia/sys.dylib
(unknown line)
jl_apply at
/Users/vagrant/buildbot/slave/package_osx10_9/build/src/./julia.h:1089
true_main at /Applications/Julia-dev.app/Contents/Resources/julia/bin/julia
(unknown line)
main at /Applications/Julia-dev.app/Contents/Resources/julia/bin/julia (unknown
line)
error in running finalizer: OverflowError()
==9270==
==9270== HEAP SUMMARY:
==9270== in use at exit: 31,285,496 bytes in 27,663 blocks
==9270== total heap usage: 45,398 allocs, 17,735 frees, 37,907,263 bytes
allocated
==9270==
==9270== LEAK SUMMARY:
==9270== definitely lost: 4,481,383 bytes in 1,343 blocks
==9270== indirectly lost: 87,248 bytes in 304 blocks
==9270== possibly lost: 22,416,431 bytes in 5,622 blocks
==9270== still reachable: 3,938,273 bytes in 19,632 blocks
==9270== suppressed: 362,161 bytes in 762 blocks
==9270== Rerun with --leak-check=full to see details of leaked memory
==9270==
==9270== For counts of detected and suppressed errors, rerun with: -v
==9270== Use --track-origins=yes to see where uninitialised values come from
==9270== ERROR SUMMARY: 366 errors from 2 contexts (suppressed: 0 from 0)
