Re: [julia-users] Memory allocation issue when using external packages in module

[Yichao Yu]

On Fri, Oct 14, 2016 at 4:14 AM, Jan  wrote:

> Thanks for the hint. That seems to be the reason!
>
> I have a couple of follow up questions so that I learn more about Julia.
> Would be nice if someone takes a couple of minutes to educate me.
>
> I found a simple example reproducing my issue:
>
> module FooBar
>
> # Including BlackBoxOptim causes type instability for exponents
>
> using BlackBoxOptim
>
> # Multiplying instead of exponentiating works fine, even when using
> BlackBoxOptim
>
> f(a::Float64, b::Float64) = a^b
>
> end
>
> If I run @code_warntype with and without BlackBoxOptim for the code above,
> it is clear that it causes the type instability.
>
> With BlackBoxOptim:
>
>
> 
>
>
> Without BlackBoxOptim:
>
>
>
> 
>
>
> For my real code, the @code_warntype macro produces identical results
> whether I use BlackBoxOptim or not even though one is type unstable.
>
> *Question 1 + related ones :) :* Is there an alternative way to check
> type instability which is more detailed but still halfway easily readable?
> Would it be possible to detect the issue with @code_llvm or similar? I
> tried to write @code_llvm to a file since the output is very long, but
> never succeeded. Is this possible?
>

code_warntype is as much detail as you can get about type instability. This
is a compiler bug, which is likely why it might looks non-obvious from the
output.
I strongly recommend **against** using code_llvm to check type stability
unless you are really familiar with llvm IR. It indeed gives you more
(lower level) information but rarely (never) about why type instability
happens.


>
> *Question 2:* When the issue 18465 is fixed, will that end up in Julia
> 0.5.1 or is a fixed version available already before that somewhere?
>

The source with the fix will be on the release-0.5 branch so you can
compile it yourself. Ideally you don't have to wait too long before it is
released though.


>
> Many thanks for any help. I really like to work with Julia so some input
> would be highly appreciated.
>
> Jan
>
>
>

Re: [julia-users] Memory allocation issue when using external packages in module

[Jan]

Thanks for the hint. That seems to be the reason!

I have a couple of follow up questions so that I learn more about Julia. 
Would be nice if someone takes a couple of minutes to educate me.

I found a simple example reproducing my issue:

module FooBar

# Including BlackBoxOptim causes type instability for exponents

using BlackBoxOptim

# Multiplying instead of exponentiating works fine, even when using 
BlackBoxOptim

f(a::Float64, b::Float64) = a^b

end

If I run @code_warntype with and without BlackBoxOptim for the code above, 
it is clear that it causes the type instability.

With BlackBoxOptim:




Without BlackBoxOptim:





For my real code, the @code_warntype macro produces identical results 
whether I use BlackBoxOptim or not even though one is type unstable.

*Question 1 + related ones :) :* Is there an alternative way to check type 
instability which is more detailed but still halfway easily readable? Would 
it be possible to detect the issue with @code_llvm or similar? I tried to 
write @code_llvm to a file since the output is very long, but never 
succeeded. Is this possible?

*Question 2:* When the issue 18465 is fixed, will that end up in Julia 
0.5.1 or is a fixed version available already before that somewhere?

Many thanks for any help. I really like to work with Julia so some input 
would be highly appreciated.

Jan

Re: [julia-users] Memory allocation issue when using external packages in module

[Yichao Yu]

Likely https://github.com/JuliaLang/julia/issues/18465

On Thu, Oct 13, 2016 at 11:56 AM, Jan  wrote:

> Hi!
>
> I´ve tried to reduce my problem to a smaller one, but I didn´t succeed.
> However, I hope someone can help me even though the problem description is
> a bit long and involves github. To summerize my problem: when I include one
> external package (BlackBoxOptim) in my module one of my functions starts to
> allocate much more memory than when omitting BlackBoxOptim.
>
> I´ve started to develop a small package for hydrological modelling:
>
> https://github.com/jmgnve/Vann
>
> I use a couple of other packages in my module (BlackBoxOptim and
> Distributions). I have simply added them like this:
>
> module Vann
>
> using BlackBoxOptim
> using Distributions
>
> other stuff
>
> end
>
> I have one function in my package called "hydro_model" which I test using
> this script:
>
> https://github.com/jmgnve/Vann/blob/master/test/hbv_tests.jl
>
> The function allocates this much memory:
>
>
> 
>
>
> If I do not include the BlackBoxOptim package in my main module (Vann.jl),
> the same function call allocates this much memory:
>
>
>
> 
>
>
> The "hydro_model"-function does not use any of the BlackBoxOptim stuff, at
> least not intentionally. The "hydro_model"-function is stored here:
>
>
> https://github.com/jmgnve/Vann/blob/master/src/hydro_hbv.jl
>
>
> Perhaps someone directly spots the issue. Am I using some name in the
> "hydro_model"-function that also exist in the BlackBoxOptim package? How
> can I analyze these kind of memory problems efficiently? The output from
> @profile is hard to understand.
>
>
> Thanks for any help.
> Jan
>
>
> *Version info:*
>
>
> *Julia Version 0.5.0*
>
> *Commit 3c9d753 (2016-09-19 18:14 UTC)*
>
> *Platform Info:*
>
> *  System: NT (x86_64-w64-mingw32)*
>
> *  CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz*
>
> *  WORD_SIZE: 64*
>
> *  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH N*
>
> *  LAPACK: libopenblas64_*
>
> *  LIBM: libopenlibm*
>
> *  LLVM: libLLVM-3.7.1 (ORCJIT, sandybridge)*
>

[julia-users] Memory allocation issue when using external packages in module

[Jan]

Hi!

I´ve tried to reduce my problem to a smaller one, but I didn´t succeed. 
However, I hope someone can help me even though the problem description is 
a bit long and involves github. To summerize my problem: when I include one 
external package (BlackBoxOptim) in my module one of my functions starts to 
allocate much more memory than when omitting BlackBoxOptim.

I´ve started to develop a small package for hydrological modelling:

https://github.com/jmgnve/Vann

I use a couple of other packages in my module (BlackBoxOptim and 
Distributions). I have simply added them like this:

module Vann

using BlackBoxOptim
using Distributions

other stuff

end

I have one function in my package called "hydro_model" which I test using 
this script:

https://github.com/jmgnve/Vann/blob/master/test/hbv_tests.jl

The function allocates this much memory:




If I do not include the BlackBoxOptim package in my main module (Vann.jl), 
the same function call allocates this much memory:





The "hydro_model"-function does not use any of the BlackBoxOptim stuff, at 
least not intentionally. The "hydro_model"-function is stored here:


https://github.com/jmgnve/Vann/blob/master/src/hydro_hbv.jl


Perhaps someone directly spots the issue. Am I using some name in the 
"hydro_model"-function that also exist in the BlackBoxOptim package? How 
can I analyze these kind of memory problems efficiently? The output from 
@profile is hard to understand.


Thanks for any help.
Jan


*Version info:*


*Julia Version 0.5.0*

*Commit 3c9d753 (2016-09-19 18:14 UTC)*

*Platform Info:*

*  System: NT (x86_64-w64-mingw32)*

*  CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz*

*  WORD_SIZE: 64*

*  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH N*

*  LAPACK: libopenblas64_*

*  LIBM: libopenlibm*

*  LLVM: libLLVM-3.7.1 (ORCJIT, sandybridge)*