Re: [julia-users] Re: map() vs broadcast()

[Kristoffer Carlsson]

This is not a new issue.

You are simply bumping into the problem that passing functions as arguments 
incur a cost every time the function is called.

If you want to compare it with map! in base you should do the following:

function mape4a(f, A, F)

tmp = similar(A)
for i in eachindex(A)
tmp[i] = f(A[i], F[i])
end

100 * sumabs(tmp) / length(A)
end


@time mape4a(_f A,F)
 0.348988 seconds (20.00 M allocations: 343.323 MB, 8.25% gc time)

There are plans on fixing this, 
see https://github.com/JuliaLang/julia/pull/13412

On Friday, October 23, 2015 at 10:58:06 AM UTC+2, Ján Dolinský wrote:
>
> versioninfo()
> Julia Version 0.4.0
> Commit 0ff703b* (2015-10-08 06:20 UTC)
> Platform Info:
>   System: Linux (x86_64-linux-gnu)
>   CPU: Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz
>   WORD_SIZE: 64
>   BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell
> )
>   LAPACK: liblapack.so.3
>   LIBM: libopenlibm
>   LLVM: libLLVM-3.3
>
> Hi Milan,
>
> The above is the versioninfo() output. I am exploring this further, using 
> map() instead of map!() give me 3 time 5 million allocations as opposed to 
> map!() with 4 times 5 million allocations. The "for" cycle in either map or 
> map!() should not allocate that much memory.  See my devectorized example 
> in the previous post.
>
> Shall I file an issue, please advise me on how to do it. In general, I 
> think map() and broadcast() should have about the same performance in the 
> example given in the beginning of this thread.
>
> Thanks,
> Jan
>
> Dňa piatok, 23. októbra 2015 10:44:01 UTC+2 Milan Bouchet-Valat 
> napísal(-a):
>>
>> This sounds suspicious to me. If you can file an issue with a 
>> reproducible example, you'll soon get feedback about what's going on 
>> here. 
>>
>> Please report the output of versioninfo() there too. I assume this is 
>> on 0.4? 
>>
>>
>> Regards 
>>
>> Le vendredi 23 octobre 2015 à 00:42 -0700, Ján Dolinský a écrit : 
>> > ## 2 argument 
>> > function map!{F}(f::F, dest::AbstractArray, A::AbstractArray, 
>> > B::AbstractArray) 
>> > for i = 1:length(A) 
>> > dest[i] = f(A[i], B[i]) 
>> > end 
>> > return dest 
>> > end 
>> > 
>> > The above is the map!() implementation in abstractarray.jl. Should it 
>> > return "dest" if it is an in-place function ? Is there any 
>> > fundamental difference between my mape4a() and map!() in 
>> > abstractarray.jl ? 
>> > 
>> > Thanks, 
>> > Jan 
>> > 
>> > Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(-a): 
>> > > Hi Glen, 
>> > > 
>> > > Thanks for the investigation. I am afraid the for loop in map!() is 
>> > > not the source of the issue. Consider the folowing: 
>> > > 
>> > > _f(a,f) = (a - f) / a 
>> > > 
>> > > function mape4(A, F) 
>> > > # A - actual target values 
>> > > # F - forecasts (model estimations) 
>> > > 
>> > >   tmp = similar(A) 
>> > >   map!(_f, tmp, A, F) 
>> > >   100 * sumabs(tmp) / length(A) 
>> > > 
>> > > end 
>> > > 
>> > > function mape4a(A, F) 
>> > > 
>> > > tmp = similar(A) 
>> > > for i in eachindex(A) 
>> > > tmp[i] = _f(A[i], F[i]) 
>> > > end 
>> > > 100 * sumabs(tmp) / length(A) 
>> > > end 
>> > > 
>> > > @time mape4(A,F) 
>> > >   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time) 
>> > > 832.852597807525 
>> > > 
>> > > @time mape4a(A,F) 
>> > >   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time) 
>> > > 832.852597807525 
>> > > 
>> > > The for loop in mape4a() does not do 4 * 5 milion allocations, 
>> > > neither should do the loop in map!(). Is this possibly a bug ? 
>> > > 
>> > > Thanks, 
>> > > Jan 
>> > > 
>> > > Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a): 
>> > > > I'm uncertain, but I think I may have figured out what's going 
>> > > > on. 
>> > > > 
>> > > > The hint lies in the number of allocations - map! has 20 million 
>> > > > allocations, while broadcast! has just 5. So I had a look at how 
>> > > > the two functions are implemented. 
>> > > > 
>> > > > map! is implemented in perhaps the simplest way you can think of 
>> > > > - for i=1:length(A) dest[i]=f(A[i],B[i]); end - which means that 
>> > > > it has to store four values per iteration - i, A[i], B[i], and 
>> > > > f(A[i],B[i]). Thus, 4 times 5 million allocations. 
>> > > > 
>> > > > broadcast! is using a cache to store values, instead, and I 
>> > > > believe it's generating instructions using a macro instead of a 
>> > > > regular loop, thus avoiding the assignments for i. As such, it 
>> > > > doesn't need to store anything except for the initial caches, and 
>> > > > after that it just overwrites the existing values. Unfortunately, 
>> > > > that's as much as I can figure out from broadcast!, because it 
>> > > > uses a lot of macros and a lot of relatively opaque structure. 
>> > > > 
>> > > > I'm also not entirely sure how it avoids the assignments 
>> > > > necessary in the function call. 
>> > > > 
>> > > > On Friday, 23 October 

Re: [julia-users] pop!(set, key, default)

[Matt]

If v is in s, I wanted first to pop v out of s out and second increase a 
counter. But you're right I can use delete! and then compare length if 
needed.

On Friday, October 23, 2015 at 12:48:08 PM UTC-4, Stefan Karpinski wrote:
>
> I'd rather have it be correct than incorrect and fast so I fixed it first. 
> I also just don't understand the rationale for the method existing in the 
> first place. If you pop something from a set you should get that value 
> back. What use is supplying a default? If you just want to remove the value 
> unconditionally, you can do delete!(s, v).
>
> On Fri, Oct 23, 2015 at 12:38 PM, Matt  
> wrote:
>
>> Thanks! 
>> I thought it was a more efficient version than 
>> if key in x
>>  pop!(x, key)
>> end
>> So yeah this method is not really useful if it is actually the same.
>>
>> I'm curious: isn't there some inefficiency in first testing for the key 
>> and then popping it out?
>>
>> On Friday, October 23, 2015 at 11:32:42 AM UTC-4, Stefan Karpinski wrote:
>>>
>>> Fixed: 
>>> https://github.com/JuliaLang/julia/commit/4164572c53b7c1ddb104a196d87558576031
>>>
>>> On Fri, Oct 23, 2015 at 10:51 AM, Stefan Karpinski >> > wrote:
>>>
 It's also a little unclear to me what this method is useful for.

 On Fri, Oct 23, 2015 at 8:53 AM, Stefan Karpinski  wrote:

> Ah, this is a bug in that pop! method. I'm working on a fix.
>
> On Fri, Oct 23, 2015 at 8:42 AM, Matt  wrote:
>
>> I want to check if a key is in a set, and pop! it if it is the case.
>>
>>
>> 1. In order to do that, I can rely on a try / catch statement (i.e. 
>> try pop!(x, key)) or a default key (i.e. pop!(x, key, nothing)). Is one 
>> preferable to the other in term of speed?
>>
>> 2.. pop!(x, key, nothing) returns nothing even if key is in x. Is 
>> this expected?
>>
>>
>> nothing == pop!(Set(1:2), 2, nothing)
>>
>> true
>>
>>
>>
>

>>>
>

Re: [julia-users] Re: map() vs broadcast()

[Milan Bouchet-Valat]

This sounds suspicious to me. If you can file an issue with a
reproducible example, you'll soon get feedback about what's going on
here.

Please report the output of versioninfo() there too. I assume this is
on 0.4?


Regards

Le vendredi 23 octobre 2015 à 00:42 -0700, Ján Dolinský a écrit :
> ## 2 argument
> function map!{F}(f::F, dest::AbstractArray, A::AbstractArray,
> B::AbstractArray)
> for i = 1:length(A)
> dest[i] = f(A[i], B[i])
> end
> return dest
> end
> 
> The above is the map!() implementation in abstractarray.jl. Should it
> return "dest" if it is an in-place function ? Is there any
> fundamental difference between my mape4a() and map!() in
> abstractarray.jl ?
> 
> Thanks,
> Jan
> 
> Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(-a):
> > Hi Glen,
> > 
> > Thanks for the investigation. I am afraid the for loop in map!() is
> > not the source of the issue. Consider the folowing:
> > 
> > _f(a,f) = (a - f) / a
> > 
> > function mape4(A, F)
> > # A - actual target values
> > # F - forecasts (model estimations)
> > 
> >   tmp = similar(A)
> >   map!(_f, tmp, A, F)
> >   100 * sumabs(tmp) / length(A)
> > 
> > end
> > 
> > function mape4a(A, F)
> > 
> > tmp = similar(A)
> > for i in eachindex(A)
> > tmp[i] = _f(A[i], F[i])
> > end
> > 100 * sumabs(tmp) / length(A)
> > end
> > 
> > @time mape4(A,F)
> >   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time)
> > 832.852597807525
> > 
> > @time mape4a(A,F)
> >   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time)
> > 832.852597807525
> > 
> > The for loop in mape4a() does not do 4 * 5 milion allocations,
> > neither should do the loop in map!(). Is this possibly a bug ?
> > 
> > Thanks,
> > Jan
> > 
> > Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a):
> > > I'm uncertain, but I think I may have figured out what's going
> > > on.
> > > 
> > > The hint lies in the number of allocations - map! has 20 million
> > > allocations, while broadcast! has just 5. So I had a look at how
> > > the two functions are implemented.
> > > 
> > > map! is implemented in perhaps the simplest way you can think of 
> > > - for i=1:length(A) dest[i]=f(A[i],B[i]); end - which means that
> > > it has to store four values per iteration - i, A[i], B[i], and
> > > f(A[i],B[i]). Thus, 4 times 5 million allocations.
> > > 
> > > broadcast! is using a cache to store values, instead, and I
> > > believe it's generating instructions using a macro instead of a
> > > regular loop, thus avoiding the assignments for i. As such, it
> > > doesn't need to store anything except for the initial caches, and
> > > after that it just overwrites the existing values. Unfortunately,
> > > that's as much as I can figure out from broadcast!, because it
> > > uses a lot of macros and a lot of relatively opaque structure.
> > > 
> > > I'm also not entirely sure how it avoids the assignments
> > > necessary in the function call.
> > > 
> > > On Friday, 23 October 2015 01:54:14 UTC+10, Ján Dolinský wrote:
> > > > Hi,
> > > > 
> > > > I am exploring Julia's map() and broadcast() functions. I did a
> > > > simple implementation of MAPE (mean absolute percentage error)
> > > > using broadcast() and map(). Interestingly, the difference in
> > > > performance was huge.
> > > > 
> > > > A = rand(5_000_000)
> > > > F = rand(5_000_000)
> > > > 
> > > > _f(a,f) = (a - f) / a
> > > > 
> > > > function mape3(A, F)
> > > > # A - actual target values
> > > > # F - forecasts (model estimations)
> > > > 
> > > >   tmp = similar(A)
> > > >   broadcast!(_f, tmp, A, F)
> > > >   100 * sumabs(tmp) / length(A)
> > > > 
> > > > end
> > > > 
> > > > function mape4(A, F)
> > > > # A - actual target values
> > > > # F - forecasts (model estimations)
> > > > 
> > > >   tmp = similar(A)
> > > >   map!(_f, tmp, A, F)
> > > >   100 * sumabs(tmp) / length(A)
> > > > 
> > > > end
> > > > 
> > > > @time mape3(A,F) # after JIT warm-up
> > > >   0.038686 seconds (8 allocations: 38.147 MB, 2.25% gc time)
> > > > 876.4813057521973
> > > > 
> > > > @time mape4(A,F) # after JIT warm-up
> > > >   0.457771 seconds (20.00 M allocations: 343.323 MB, 11.29% gc
> > > > time)
> > > > 876.4813057521973
> > > > 
> > > > I wonder why map() is so much slower ?
> > > > 
> > > > Thanks,
> > > > Jan
> > > > 

Re: [julia-users] Re: map() vs broadcast()

[Ján Dolinský]

versioninfo()
Julia Version 0.4.0
Commit 0ff703b* (2015-10-08 06:20 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Hi Milan,

The above is the versioninfo() output. I am exploring this further, using 
map() instead of map!() give me 3 time 5 million allocations as opposed to 
map!() with 4 times 5 million allocations. The "for" cycle in either map or 
map!() should not allocate that much memory.  See my devectorized example 
in the previous post.

Shall I file an issue, please advise me on how to do it. In general, I 
think map() and broadcast() should have about the same performance in the 
example given in the beginning of this thread.

Thanks,
Jan

Dňa piatok, 23. októbra 2015 10:44:01 UTC+2 Milan Bouchet-Valat napísal(-a):
>
> This sounds suspicious to me. If you can file an issue with a 
> reproducible example, you'll soon get feedback about what's going on 
> here. 
>
> Please report the output of versioninfo() there too. I assume this is 
> on 0.4? 
>
>
> Regards 
>
> Le vendredi 23 octobre 2015 à 00:42 -0700, Ján Dolinský a écrit : 
> > ## 2 argument 
> > function map!{F}(f::F, dest::AbstractArray, A::AbstractArray, 
> > B::AbstractArray) 
> > for i = 1:length(A) 
> > dest[i] = f(A[i], B[i]) 
> > end 
> > return dest 
> > end 
> > 
> > The above is the map!() implementation in abstractarray.jl. Should it 
> > return "dest" if it is an in-place function ? Is there any 
> > fundamental difference between my mape4a() and map!() in 
> > abstractarray.jl ? 
> > 
> > Thanks, 
> > Jan 
> > 
> > Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(-a): 
> > > Hi Glen, 
> > > 
> > > Thanks for the investigation. I am afraid the for loop in map!() is 
> > > not the source of the issue. Consider the folowing: 
> > > 
> > > _f(a,f) = (a - f) / a 
> > > 
> > > function mape4(A, F) 
> > > # A - actual target values 
> > > # F - forecasts (model estimations) 
> > > 
> > >   tmp = similar(A) 
> > >   map!(_f, tmp, A, F) 
> > >   100 * sumabs(tmp) / length(A) 
> > > 
> > > end 
> > > 
> > > function mape4a(A, F) 
> > > 
> > > tmp = similar(A) 
> > > for i in eachindex(A) 
> > > tmp[i] = _f(A[i], F[i]) 
> > > end 
> > > 100 * sumabs(tmp) / length(A) 
> > > end 
> > > 
> > > @time mape4(A,F) 
> > >   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time) 
> > > 832.852597807525 
> > > 
> > > @time mape4a(A,F) 
> > >   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time) 
> > > 832.852597807525 
> > > 
> > > The for loop in mape4a() does not do 4 * 5 milion allocations, 
> > > neither should do the loop in map!(). Is this possibly a bug ? 
> > > 
> > > Thanks, 
> > > Jan 
> > > 
> > > Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a): 
> > > > I'm uncertain, but I think I may have figured out what's going 
> > > > on. 
> > > > 
> > > > The hint lies in the number of allocations - map! has 20 million 
> > > > allocations, while broadcast! has just 5. So I had a look at how 
> > > > the two functions are implemented. 
> > > > 
> > > > map! is implemented in perhaps the simplest way you can think of 
> > > > - for i=1:length(A) dest[i]=f(A[i],B[i]); end - which means that 
> > > > it has to store four values per iteration - i, A[i], B[i], and 
> > > > f(A[i],B[i]). Thus, 4 times 5 million allocations. 
> > > > 
> > > > broadcast! is using a cache to store values, instead, and I 
> > > > believe it's generating instructions using a macro instead of a 
> > > > regular loop, thus avoiding the assignments for i. As such, it 
> > > > doesn't need to store anything except for the initial caches, and 
> > > > after that it just overwrites the existing values. Unfortunately, 
> > > > that's as much as I can figure out from broadcast!, because it 
> > > > uses a lot of macros and a lot of relatively opaque structure. 
> > > > 
> > > > I'm also not entirely sure how it avoids the assignments 
> > > > necessary in the function call. 
> > > > 
> > > > On Friday, 23 October 2015 01:54:14 UTC+10, Ján Dolinský wrote: 
> > > > > Hi, 
> > > > > 
> > > > > I am exploring Julia's map() and broadcast() functions. I did a 
> > > > > simple implementation of MAPE (mean absolute percentage error) 
> > > > > using broadcast() and map(). Interestingly, the difference in 
> > > > > performance was huge. 
> > > > > 
> > > > > A = rand(5_000_000) 
> > > > > F = rand(5_000_000) 
> > > > > 
> > > > > _f(a,f) = (a - f) / a 
> > > > > 
> > > > > function mape3(A, F) 
> > > > > # A - actual target values 
> > > > > # F - forecasts (model estimations) 
> > > > > 
> > > > >   tmp = similar(A) 
> > > > >   broadcast!(_f, tmp, A, F) 
> > > > >   100 * sumabs(tmp) / length(A) 
> > > > > 
> > > > > end 
> > > > > 
> > > > > function mape4(A, F) 
> > > > > # A - 

Re: [julia-users] Re: map() vs broadcast()

[Milan Bouchet-Valat]

Le vendredi 23 octobre 2015 à 01:58 -0700, Ján Dolinský a écrit :
> versioninfo()
> Julia Version 0.4.0
> Commit 0ff703b* (2015-10-08 06:20 UTC)
> Platform Info:
>   System: Linux (x86_64-linux-gnu)
>   CPU: Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz
>   WORD_SIZE: 64
>   BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY
> Haswell)
>   LAPACK: liblapack.so.3
>   LIBM: libopenlibm
>   LLVM: libLLVM-3.3
> 
> Hi Milan,
> 
> The above is the versioninfo() output. I am exploring this further,
> using map() instead of map!() give me 3 time 5 million allocations as
> opposed to map!() with 4 times 5 million allocations. The "for" cycle
> in either map or map!() should not allocate that much memory.  See my
> devectorized example in the previous post.
> 
> Shall I file an issue, please advise me on how to do it. In general,
> I think map() and broadcast() should have about the same performance
> in the example given in the beginning of this thread.
Just present the different functions you wrote below and the
timings/allocations, should be enough.


Regards

> Thanks,
> Jan
> 
> Dňa piatok, 23. októbra 2015 10:44:01 UTC+2 Milan Bouchet-Valat
> napísal(-a):
> > This sounds suspicious to me. If you can file an issue with a 
> > reproducible example, you'll soon get feedback about what's going
> > on 
> > here. 
> > 
> > Please report the output of versioninfo() there too. I assume this
> > is 
> > on 0.4? 
> > 
> > 
> > Regards 
> > 
> > Le vendredi 23 octobre 2015 à 00:42 -0700, Ján Dolinský a écrit : 
> > > ## 2 argument 
> > > function map!{F}(f::F, dest::AbstractArray, A::AbstractArray, 
> > > B::AbstractArray) 
> > > for i = 1:length(A) 
> > > dest[i] = f(A[i], B[i]) 
> > > end 
> > > return dest 
> > > end 
> > > 
> > > The above is the map!() implementation in abstractarray.jl.
> > Should it 
> > > return "dest" if it is an in-place function ? Is there any 
> > > fundamental difference between my mape4a() and map!() in 
> > > abstractarray.jl ? 
> > > 
> > > Thanks, 
> > > Jan 
> > > 
> > > Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(
> > -a): 
> > > > Hi Glen, 
> > > > 
> > > > Thanks for the investigation. I am afraid the for loop in
> > map!() is 
> > > > not the source of the issue. Consider the folowing: 
> > > > 
> > > > _f(a,f) = (a - f) / a 
> > > > 
> > > > function mape4(A, F) 
> > > > # A - actual target values 
> > > > # F - forecasts (model estimations) 
> > > > 
> > > >   tmp = similar(A) 
> > > >   map!(_f, tmp, A, F) 
> > > >   100 * sumabs(tmp) / length(A) 
> > > > 
> > > > end 
> > > > 
> > > > function mape4a(A, F) 
> > > > 
> > > > tmp = similar(A) 
> > > > for i in eachindex(A) 
> > > > tmp[i] = _f(A[i], F[i]) 
> > > > end 
> > > > 100 * sumabs(tmp) / length(A) 
> > > > end 
> > > > 
> > > > @time mape4(A,F) 
> > > >   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc
> > time) 
> > > > 832.852597807525 
> > > > 
> > > > @time mape4a(A,F) 
> > > >   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time) 
> > > > 832.852597807525 
> > > > 
> > > > The for loop in mape4a() does not do 4 * 5 milion allocations, 
> > > > neither should do the loop in map!(). Is this possibly a bug ? 
> > > > 
> > > > Thanks, 
> > > > Jan 
> > > > 
> > > > Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(
> > -a): 
> > > > > I'm uncertain, but I think I may have figured out what's
> > going 
> > > > > on. 
> > > > > 
> > > > > The hint lies in the number of allocations - map! has 20
> > million 
> > > > > allocations, while broadcast! has just 5. So I had a look at
> > how 
> > > > > the two functions are implemented. 
> > > > > 
> > > > > map! is implemented in perhaps the simplest way you can think
> > of 
> > > > > - for i=1:length(A) dest[i]=f(A[i],B[i]); end - which means
> > that 
> > > > > it has to store four values per iteration - i, A[i], B[i],
> > and 
> > > > > f(A[i],B[i]). Thus, 4 times 5 million allocations. 
> > > > > 
> > > > > broadcast! is using a cache to store values, instead, and I 
> > > > > believe it's generating instructions using a macro instead of
> > a 
> > > > > regular loop, thus avoiding the assignments for i. As such,
> > it 
> > > > > doesn't need to store anything except for the initial caches,
> > and 
> > > > > after that it just overwrites the existing values.
> > Unfortunately, 
> > > > > that's as much as I can figure out from broadcast!, because
> > it 
> > > > > uses a lot of macros and a lot of relatively opaque
> > structure. 
> > > > > 
> > > > > I'm also not entirely sure how it avoids the assignments 
> > > > > necessary in the function call. 
> > > > > 
> > > > > On Friday, 23 October 2015 01:54:14 UTC+10, Ján Dolinský
> > wrote: 
> > > > > > Hi, 
> > > > > > 
> > > > > > I am exploring Julia's map() and broadcast() functions. I
> > did a 
> > > > > > simple implementation of MAPE (mean absolute percentage
> > error) 
> > > > > > using broadcast() and map(). 

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Christoph Ortner]

If I write into a view, does it change the original array? 

Christoph

[julia-users] Re: map() vs broadcast()

[DNF]

I guess this is mostly about the behaviour of map!, but for avoiding 
allocations I think this makes more sense:

function mape4b(A, F)
tmp = zero(eltype(A))
N = length(A)
for i in 1:N
tmp += abs(_f(A[i], F[i]))
end
return 100 * tmp / N
end

For some reason, the time macro doesn't give me any number for allocation 
in mape4b, but it's 20-25% faster than mape4a and almost twice as fast as 
the broadcast! version on my computer.

I tried using @inbounds but that had no effect whatsoever.

[julia-users] are array slices views in 0.4?

[Neal Becker]

One feature I've been eager to see is that array slices become views, which 
I think was targeted for 0.4.  Is this correct?  I hate to see un-needed 
copying.

Re: [julia-users] Julia and Object-Oriented Programming

[Brendan Tracey]

On Friday, October 23, 2015 at 7:38:37 AM UTC-6, Abe Schneider wrote:
>
> An OO approach is really just specifying an interface in a formal manner. 
> The second you write any type of interface, you always risk making a choice 
> that will haunt you down the road. I don't see the difference between:
>
> class Foo {
>   float getX() { ... }
>   float getY() { ... }
> }
>
> and:
>
> type Foo { ... }
> function getX(f::Foo) -> float { ... }
> function getY(f::Foo) -> float { ... }
>
> except that an instance of `foo` is being passed implicitly in the first 
> case. To that extent, I would say Julia OO (i.e. on the dispatching). Where 
> it is not OO is in the layout of the data.
>

In some  languages there is no difference. In Go, a method is exactly a 
function that can be called with a special syntax. One can do

c := mat64.NewDense(5, 5, nil)
c.Mul(a, b)
 
or one can do
(*mat64.Dense).Mul(c, a, b)

There aren't many cases where one would actually want to do the latter, but 
it makes the definition of behavior clear.

I think the difference in Julia would be the interaction with multiple 
dispatch. In the former case, it seems like getX would live in its own 
namespace (only accessible through the Foo class), while in the latter case 
it would add to the many definitions of getX.


It is true that with a more OO language, because of its formalism, you run 
> the risk of declaring variable types in a parent class that might not be 
> correct in the child classes, so some planning is required. However, what 
> you gain is better code in the long run. For example, I might have:
>
> abstract Baz
>
> type Foo <: Baz {
>   x::Float
> }
>
> type Bar <: Baz {
>   x::Float
> }
>
> and an interface:
>
> getX{T <: Baz}(b::T) -> Float = b.x
>
>
> which works fine, except maybe someone else comes along and writes:
>
> type Blob <: Baz {
>   myX::Float
> }
>
> Now to fix your interface, you have to write a separate `getX` for `Blob`. 
> This might not seem like a big deal, except you might not catch this issue 
> until run time (I don't think there is a way a static-checker could 
> identify the problem). Imagine a large library or base of code, with many 
> people working on the code, and you suddenly are exposed to a large number 
> of issues.
>

This seems more like an issue of compiler vs. no compiler issue than one 
with OO vs. multiple dispatch.
 
You can do OO without inheritance (or OO like things if your definition of 
OO includes inheritance). In Go, one would define 

type GetXer interface {
 GetX() float64
}

I could then write a function
func SquareX(g GetXer) float64 {
 v := g.GetX()
 return v * v
}

Now, any type that has a GetX method can be passed to SquareX. There is no 
inheritance necessary to make this happen.

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Tim Holy]

My point is it's just notation. `M[:,2]` is notation for `getindex(M, :, 2)`, 
which returns a copy. `slice(M, :, 2)` returns a view. In julia 0.5, `M[:,2]` 
will effectively become notation for `slice(M, :, 2)`.

So we're talking about a difference of 6 characters. I know syntax is 
important, and that as heavily as arrays are used, 6 characters may not be 
trivial. (More importantly, the lack of bounds checking is not to be 
dismissed.) Still, it's worth distinguishing between "how many keystrokes do I 
have to type?" and "what functionality exists today?"---those are two very 
different questions.

--Tim

On Friday, October 23, 2015 11:48:31 AM Neal Becker wrote:
> Tim Holy wrote:
> > This will be a ridiculous charicature, but this issue has come up so many
> > times that I think a little charicaturization is timely.
> > 
> > With regards to arrays and views, the "big change" in julia 0.5 will
> > 
> > essentially be a 1-line change:
> >  getindex(A, indexes...) = slice(A, indexes...)
> > 
> > Since slice is already in base, you can use it today :-).
> > 
> > There's much more to it than that: SubArrays don't always check bounds, so
> > currently this is At Your Own Risk. There are some difficulties in
> > reshaping SubArrays.
> > 
> > But if the fundamental question is, "can I have performant and very
> > flexible views?" the answer is: you already have them.
> > 
> > Best,
> > --Tim
> 
> To be more precise, I'm coming from python/numpy.  There, a native slice:
> M[1:,2:] for example
> returns a view, not a copy. As does a strided view:
> M[::2]
>  I like this design.  Does julia do this now, or does it plan to in the
> future?

[julia-users] Re: what is best practice for allowing pre-'using' selection of a module-level setting

[Jeffrey Sarnoff]

Thank you -- clearly given.


On Friday, October 23, 2015 at 12:07:07 PM UTC-4, Steven G. Johnson wrote:
>
>
>
> On Friday, October 23, 2015 at 8:37:58 AM UTC-4, Jeffrey Sarnoff wrote:
>>
>> I want to allow users of my Float12x module the choice of faster or more 
>> precise trig
>>(at the moment, 103bits over 0..2pi is ~30% faster than 106bits; the 
>> more precise version also handles larger arguments).
>> The choice needs to be made before the using statement, as it governs the 
>> inclusion of either one of two small source files.
>>
>
> It would be better to refactor your code so that the choice can be made 
> dynamically, after the module is loaded.
>
> However, if it is really impossible to do this dynamically, then this kind 
> of "static" choice would normally be made at Pkg.build time, and a common 
> mechanism for specifying it would be an environment variable.   The 
> advantage of using Pkg.build is that then your module can be precompiled.   
> The procedure I've used is something like:
>
> 1) create a deps/build.jl file that looks at ENV["FASTTRIG"] (or whatever) 
> and spits out:
>
>a) a deps/deps.jl file with a constant like `const FASTTRIG = true`
>
>b) a deps/FASTTRIG file to save the FASTRIG setting for the next 
> time the file is built.  This way, if your package is rebuilt on e.g. a 
> Pkg.update, you won't "forget" the user's preference.i.e. you would 
> read ENV["FASTTRIG"] by something like:
>   get(ENV, "FASTTRIG", isfile("FASTTRIG") ? 
> readchomp("FASTTRIG") : "no")
>
> 2) in your main module file, do
>
> const depfile = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")
>
> isfile(depfile) || error("Float12x not properly installed. Please run 
> Pkg.build(\"Float12x\")")
>
> include(depfile) # generated by Pkg.build("Float12x")
>
>
> and then use the value of FASTTRIG to decide how to proceed.
>
>
> 3) Tell your users to do
>
>
> ENV["FASTTRIG"] = "yes" or "no"
>
> Pkg.build("Float12x")
>
>
> when they want to change the setting.
>

[julia-users] Re: Re: are array slices views in 0.4?

[Neal Becker]

Tim Holy wrote:

> This will be a ridiculous charicature, but this issue has come up so many
> times that I think a little charicaturization is timely.
> 
> With regards to arrays and views, the "big change" in julia 0.5 will
> essentially be a 1-line change:
> 
>  getindex(A, indexes...) = slice(A, indexes...)
> 
> Since slice is already in base, you can use it today :-).
> 
> There's much more to it than that: SubArrays don't always check bounds, so
> currently this is At Your Own Risk. There are some difficulties in
> reshaping SubArrays.
> 
> But if the fundamental question is, "can I have performant and very
> flexible views?" the answer is: you already have them.
> 
> Best,
> --Tim

To be more precise, I'm coming from python/numpy.  There, a native slice:
M[1:,2:] for example
returns a view, not a copy. As does a strided view:
M[::2]
 I like this design.  Does julia do this now, or does it plan to in the 
future?

[julia-users] Re: Help on optimization problem

[Tony Kelman]

Brendan, have you used JuMP? It's primarily designed for constrained 
problems, and it works great. Don't be afraid of constraints if they help 
you express your problem in a cleaner way or with more of the structure 
preserved. For things like constant parameter values though, JuMP supports 
"fixed" variables which get sent to the solvers as if they're parameters. 
Most good constrained solvers have presolve routines that can detect 
trivial constraints like single-variable equalities and will do the 
substitution automatically, but doesn't hurt to provide information in a 
more specific way.


On Friday, October 23, 2015 at 8:03:27 AM UTC-7, Brendan Tracey wrote:
>
>
> 2. (JuMP Specific) - Should I specify my known positions as model 
>> variables with equality constraints, or just normal julia variables that 
>> show up in my objective function? 
>>
>
> Don't specify them as equality constraints. Build your function with those 
> variables removed from the optimization altogether. Perhaps that's what you 
> meant by "normal julia variables"? In any event, unconstrained optimization 
> is much more robust and mature, and you will likely get better results the 
> fewer constraints you give.  
>

[julia-users] Re: what is best practice for allowing pre-'using' selection of a module-level setting

[Jeffrey Sarnoff]

The swap involves two small functions, in either case they share the same 
names.  They service internal angle remapping.  I can give each pair's 
names a suffix, and include both cases all the time -- which would be 
helpful as this stuff should be precompiled.  What approach would put one 
or the other pair in the midst of an encompassing and encompassed chain of 
function calls?
And allow them to be swapped at runtime with a module level call?

On Friday, October 23, 2015 at 12:07:07 PM UTC-4, Steven G. Johnson wrote:
>
>
>
> On Friday, October 23, 2015 at 8:37:58 AM UTC-4, Jeffrey Sarnoff wrote:
>>
>> I want to allow users of my Float12x module the choice of faster or more 
>> precise trig
>>(at the moment, 103bits over 0..2pi is ~30% faster than 106bits; the 
>> more precise version also handles larger arguments).
>> The choice needs to be made before the using statement, as it governs the 
>> inclusion of either one of two small source files.
>>
>
> It would be better to refactor your code so that the choice can be made 
> dynamically, after the module is loaded.
>
> However, if it is really impossible to do this dynamically, then this kind 
> of "static" choice would normally be made at Pkg.build time, and a common 
> mechanism for specifying it would be an environment variable.   The 
> advantage of using Pkg.build is that then your module can be precompiled.   
> The procedure I've used is something like:
>
> 1) create a deps/build.jl file that looks at ENV["FASTTRIG"] (or whatever) 
> and spits out:
>
>a) a deps/deps.jl file with a constant like `const FASTTRIG = true`
>
>b) a deps/FASTTRIG file to save the FASTRIG setting for the next 
> time the file is built.  This way, if your package is rebuilt on e.g. a 
> Pkg.update, you won't "forget" the user's preference.i.e. you would 
> read ENV["FASTTRIG"] by something like:
>   get(ENV, "FASTTRIG", isfile("FASTTRIG") ? 
> readchomp("FASTTRIG") : "no")
>
> 2) in your main module file, do
>
> const depfile = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")
>
> isfile(depfile) || error("Float12x not properly installed. Please run 
> Pkg.build(\"Float12x\")")
>
> include(depfile) # generated by Pkg.build("Float12x")
>
>
> and then use the value of FASTTRIG to decide how to proceed.
>
>
> 3) Tell your users to do
>
>
> ENV["FASTTRIG"] = "yes" or "no"
>
> Pkg.build("Float12x")
>
>
> when they want to change the setting.
>

Re: [julia-users] Re: what is best practice for allowing pre-'using' selection of a module-level setting

[Tom Breloff]

If someone does:

ENV["FASTTRIG"] = "yes"
Pkg.build("Float12x")

and then in another session does Pkg.build(), (presumably for some other
package or update), then won't the setting be overwritten with the default
build setting?

On Fri, Oct 23, 2015 at 12:11 PM, Jeffrey Sarnoff  wrote:

> Thank you -- clearly given.
>
>
> On Friday, October 23, 2015 at 12:07:07 PM UTC-4, Steven G. Johnson wrote:
>>
>>
>>
>> On Friday, October 23, 2015 at 8:37:58 AM UTC-4, Jeffrey Sarnoff wrote:
>>>
>>> I want to allow users of my Float12x module the choice of faster or more
>>> precise trig
>>>(at the moment, 103bits over 0..2pi is ~30% faster than 106bits; the
>>> more precise version also handles larger arguments).
>>> The choice needs to be made before the using statement, as it governs
>>> the inclusion of either one of two small source files.
>>>
>>
>> It would be better to refactor your code so that the choice can be made
>> dynamically, after the module is loaded.
>>
>> However, if it is really impossible to do this dynamically, then this
>> kind of "static" choice would normally be made at Pkg.build time, and a
>> common mechanism for specifying it would be an environment variable.   The
>> advantage of using Pkg.build is that then your module can be precompiled.
>> The procedure I've used is something like:
>>
>> 1) create a deps/build.jl file that looks at ENV["FASTTRIG"] (or
>> whatever) and spits out:
>>
>>a) a deps/deps.jl file with a constant like `const FASTTRIG = true`
>>
>>b) a deps/FASTTRIG file to save the FASTRIG setting for the next
>> time the file is built.  This way, if your package is rebuilt on e.g. a
>> Pkg.update, you won't "forget" the user's preference.i.e. you would
>> read ENV["FASTTRIG"] by something like:
>>   get(ENV, "FASTTRIG", isfile("FASTTRIG") ?
>> readchomp("FASTTRIG") : "no")
>>
>> 2) in your main module file, do
>>
>> const depfile = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")
>>
>> isfile(depfile) || error("Float12x not properly installed. Please run
>> Pkg.build(\"Float12x\")")
>>
>> include(depfile) # generated by Pkg.build("Float12x")
>>
>>
>> and then use the value of FASTTRIG to decide how to proceed.
>>
>>
>> 3) Tell your users to do
>>
>>
>> ENV["FASTTRIG"] = "yes" or "no"
>>
>> Pkg.build("Float12x")
>>
>>
>> when they want to change the setting.
>>
>

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Tom Breloff]

On this note, is there a good summary somewhere of the pros/cons between
using `sub` vs `slice` vs `ArrayViews.view`?

On Fri, Oct 23, 2015 at 11:57 AM, Steven G. Johnson 
wrote:

> On Friday, October 23, 2015 at 11:49:21 AM UTC-4, Neal Becker wrote:
>>
>> To be more precise, I'm coming from python/numpy.  There, a native slice:
>> M[1:,2:] for example
>> returns a view, not a copy. As does a strided view:
>> M[::2]
>>  I like this design.  Does julia do this now, or does it plan to in the
>> future?
>
>
> In 0.5, the "native" slice syntax like M[1:end,2:end], both strided and
> non-strided, will return a view.
>
> In 0.4, M[:] etc returns a copy, but you can get a view by the "slice"
> function as Tim mentioned.  So, the functionality is there now, but the
> syntax is a little less familiar.
>

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Patrick O'Leary]

On Friday, October 23, 2015 at 6:18:43 AM UTC-5, Kris De Meyer wrote:
>
> ...and then the only thing Julia will have going for it is that it's free. 
> But my cost to my employers is such that if I lose as little as 3 days a 
> year on compatibility issues, they would be better off paying for a Matlab 
> license... 
>

It's not quite that simple--if I have a problem with something in MATLAB, I 
file a bug report. The issue may (or may not) be fixed in 3-6 months, with 
the next release. If I'm lucky I catch problems in prereleases--but that's 
also time I have to spend to install the prerelease and test the fragile 
parts of the process. This isn't hypothetical; we've skipped entire 
releases which have variously broken code generation. (Full disclosure, we 
haven't seen such problems with MATLAB proper, but deployment with Coder is 
essential for what we do.)

But if something broke in Julia, there's a chance I can fix it. Free 
software matters.

Patrick

Re: [julia-users] Julia and Object-Oriented Programming

[Abe Schneider]

An OO approach is really just specifying an interface in a formal manner. 
The second you write any type of interface, you always risk making a choice 
that will haunt you down the road. I don't see the difference between:

class Foo {
  float getX() { ... }
  float getY() { ... }
}

and:

type Foo { ... }
function getX(f::Foo) -> float { ... }
function getY(f::Foo) -> float { ... }

except that an instance of `foo` is being passed implicitly in the first 
case. To that extent, I would say Julia OO (i.e. on the dispatching). Where 
it is not OO is in the layout of the data.

It is true that with a more OO language, because of its formalism, you run 
the risk of declaring variable types in a parent class that might not be 
correct in the child classes, so some planning is required. However, what 
you gain is better code in the long run. For example, I might have:

abstract Baz

type Foo <: Baz {
  x::Float
}

type Bar <: Baz {
  x::Float
}

and an interface:

getX{T <: Baz}(b::T) -> Float = b.x


which works fine, except maybe someone else comes along and writes:

type Blob <: Baz {
  myX::Float
}

Now to fix your interface, you have to write a separate `getX` for `Blob`. 
This might not seem like a big deal, except you might not catch this issue 
until run time (I don't think there is a way a static-checker could 
identify the problem). Imagine a large library or base of code, with many 
people working on the code, and you suddenly are exposed to a large number 
of issues.

This is why people advocate OOP. In the first definition of `Foo`, I know 
it will always have an `x`, and I can easily see the interface necessary to 
access it.

So, yes, an OO usually requires more work up-front. If Julia is meant to be 
purely scientific language which requires a code rewrite for real-world 
use, then I won't argue with not having a way to compose the data. However, 
in any project of medium to large complexity, it will make life much more 
difficult not having that capability.

On Thursday, October 22, 2015 at 4:00:36 PM UTC-4, vav...@uwaterloo.ca 
wrote:
>
> One way to build a code-base that everyone can share is to specify 
> interfaces to datatypes as in methods in C++ base classes.  Another way is 
> for each individual to write his/her own code, and then read everyone 
> else's code, and then meet at coffee shops and talk on the phone to get 
> things to work together.  I am claiming that the second way is better for 
> scientific software than the first way because scientific software is more 
> open-ended than, say, systems software.  I am claiming that specifying 
> interfaces from the outset can lead to prematurely locking in design 
> decisions, and that I have seen examples of this in the real world.  Keep 
> in mind that Julia is being touted as a suitable language for both the 
> initial and the later phases of scientific software development.
>
> In the event that Julia adds new OO or other interface-specifying 
> features, obviously it would be possible for a team to ignore them, at 
> least initially.  But I have also seen in previous projects that there is a 
> tendency for people (including me) to jump onto the fanciest possible 
> solution offered by the programming language to solve a software design 
> problem.
>
> -- Steve Vavasis
>
>
> On Thursday, October 22, 2015 at 12:02:21 PM UTC-4, g wrote:
>>
>> What is the other option here? It seemed like with the OO/Julia way you 
>> are complaining about you at least have working (but slow) code handling 
>> your new polynomial type. In a case where your new type doesn't work with 
>> "obtainCoefficient", it won't 
>> work with any of your other code either. You would just have no working 
>> code with your new polynomial type. How is that better?
>>
>> But the next week, someone asks whether you can handle polynomials 
>>> specified as p(x)=det(A-x*B), where A and B are n-by-n matrices.  For 
>>> polynomials in this format, "obtainCoefficient" is expensive and would not 
>>> be regarded as a simple "getter" operation.  If many people had already 
>>> written functions invoking the 'obtainCoefficient' method, then you would 
>>> be stuck.  You would retrospectively realize that the obtainCoefficient 
>>> member function was not a good idea.  This example is typical of open-ended 
>>> scientific software projects.
>>>

[julia-users] pop!(set, key, default)

[Matt]

 

I want to check if a key is in a set, and pop! it if it is the case.


1. In order to do that, I can rely on a try / catch statement (i.e. try 
pop!(x, key)) or a default key (i.e. pop!(x, key, nothing)). Is one 
preferable to the other in term of speed?

2.. pop!(x, key, nothing) returns nothing even if key is in x. Is this 
expected?


nothing == pop!(Set(1:2), 2, nothing)

true

Re: [julia-users] Parallel loop

[amiksvi]

Thanks a lot for this hint Tim. I think this is indeed exactly what I need. 
I have implemented it more or less successfully in the sense that it works 
and computes the correct matrix:

## parallel helpers, from there: 
http://stackoverflow.com/questions/27677399/julia-how-to-copy-data-to-another-processor-in-julia

function sendto(p::Int; args...)
for (nm, val) in args
@spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
end

function sendto(ps::Vector{Int}; args...)
for p in ps
sendto(p; args...)
end
end

getfrom(p::Int, nm::Symbol; mod=Main) = fetch(@spawnat(p, getfield(mod, 
nm)))

function passobj(src::Int, target::Vector{Int}, nm::Symbol;
 from_mod=Main, to_mod=Main)
r = RemoteRef(src)
@spawnat(src, put!(r, getfield(from_mod, nm)))
for to in target
@spawnat(to, eval(to_mod, Expr(:(=), nm, fetch(r
end
nothing
end

function passobj(src::Int, target::Int, nm::Symbol; from_mod=Main, 
to_mod=Main)
passobj(src, [target], nm; from_mod=from_mod, to_mod=to_mod)
end

function passobj(src::Int, target, nms::Vector{Symbol};
 from_mod=Main, to_mod=Main)
for nm in nms
passobj(src, target, nm; from_mod=from_mod, to_mod=to_mod)
end
end

## variables

m = Int(1e3)
n = Int(1e4)
mat_b = rand(m, n)

## sequential

function compute_row_sequential(mat_b, i, n)
return mean(mat_b[:,i] .* mat_b[:,i:n], 1)
end

mat_a = zeros(n, n)

tic()
for i = 1:n
mat_a[i,i:n] = compute_row_sequential(mat_b, i, n)
end
toc()

## parallel

addprocs(3)

@everywhere function compute_row_shared!(smat_a, mat_b, irange, n)
for i in irange
smat_a[i,i:n] = mean(mat_b[:,i] .* mat_b[:,i:n], 1)
end
end

sendto(workers(), n = n)
sendto(workers(), mat_b = mat_b)

smat_a = SharedArray(Float64, (n,n), pids = workers())

tic()
@sync begin
for p in procs(smat_a)
@async begin
irange = p-1:length(procs(smat_a)):n
remotecall_wait(p, compute_row_shared!, smat_a, mat_b, irange, 
n)
end
end
end
toc()

println(mat_a == smat_a)

the last line returns true, but I tried different values of m and n and I 
could not find a case where the parallel implementation is more efficient 
than the sequential one. So obviously, either I'm doing something wrong or 
it's not the best approach for this case...

I'll keep on trying to improve the efficiency, any advice again welcome :)

Thank you.

[julia-users] Package can't be used

[midsummer . shang]

  I'm using IJulia and  command line version of Julia 0-4-4. 

  When using PyPlot in IJulia, it fails and shows the following:

LoadError: InitError: Failed to pyimport("matplotlib"): PyPlot will not work 
until you have a functioning matplotlib module.

For automated Matplotlib installation, try configuring PyCall to use the Conda 
Python distribution within Julia.  Relaunch Julia and run:
  ENV["PYTHON"]=""
  Pkg.build("PyCall")
  using PyPlot

pyimport exception was: PyError (:PyImport_ImportModule) 
ImportError('No module named matplotlib',)

during initialization of module PyPlot
while loading In[10], in expression starting on line 2

 in __init__ at /Users/yxshang/.julia/v0.4/PyPlot/src/PyPlot.jl:229
 in _require_from_serialized at loading.jl:84
 in _require_from_serialized at 
/Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib 

in require at 
/Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib, 
 

 The package is downloaded and can be find in terminal. But it can't be 
used.
 
 And some other package you clone form github also can't be load and often 
shows, for example:

LoadError: error compiling anonymous: error compiling call: could not load 
library "/Users/yxshang/.julia/v0.4/Toms566/src/lib566"


 Is it because i didn't set the proper path to the package? Then how to fix it?

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Stefan Karpinski]

Yes, my experience with upgrading large code bases to new point releases is
that no changes are required. Kris, if your experience with point releases
has been otherwise, it would be good to hear about specific issues since
those releases are strictly bug fix releases. Perhaps you are talking about
the last major upgrade from 0.2.x. to 0.3.x?

On Fri, Oct 23, 2015 at 8:25 AM, Tony Kelman  wrote:

> >  I've been developing Julia code since release v0.3.7. Unfortunately
> this isn't the first time that I lose many hours having to figure out why
> something that works in one release stops working in a subsequent one.
>
> This surprises me. 0.4.0 is the only release that was intended to be
> breaking in any meaningful way since 0.3.0. Every point release
> announcement has said "We would like to get feedback if someone has a
> working program that breaks after this upgrade." I meant that seriously, if
> an unintended breaking change was made in any of the 0.3.x series and an
> issue was reported about it, we would find a way to fix it (for example,
> there was a problem in the embedding API that led us to put out 0.3.5 on an
> accelerated schedule to fix it). The same holds for 0.4.x relative to 0.4.0.
>

Re: [julia-users] pop!(set, key, default)

[Stefan Karpinski]

Ah, this is a bug in that pop! method. I'm working on a fix.

On Fri, Oct 23, 2015 at 8:42 AM, Matt  wrote:

> I want to check if a key is in a set, and pop! it if it is the case.
>
>
> 1. In order to do that, I can rely on a try / catch statement (i.e. try
> pop!(x, key)) or a default key (i.e. pop!(x, key, nothing)). Is one
> preferable to the other in term of speed?
>
> 2.. pop!(x, key, nothing) returns nothing even if key is in x. Is this
> expected?
>
>
> nothing == pop!(Set(1:2), 2, nothing)
>
> true
>
>
>

Re: [julia-users] Julia and Object-Oriented Programming

[Abe Schneider]

Ah, okay, that is different then. What's the advantage of creating a new 
method versus copying the fields? I would imagine there is a penalty with 
each deference you have to follow in order to make that work.


I'm not familiar with Scala, so sorry if I'm telling you something you 
> know, but in go the member variables and methods are not copied over. The 
> Unicycle struct in go does not itself have those fields, the fields of the 
> struct do. In other words, defining
>
> type Unicycle struct {
> Frame
> Wheel
> Seat
> } 
>
> Is exactly like declaring 
>
> type Unicycle struct {
>Frame Frame
>Wheel Wheel
>SeatSeat
> } 
>
> except that in the former case one can call unicycle.Cushiness() , and in 
> the latter one must call unicycle.Seat.Cushiness(). In particular, if you 
> swap out one Frame in the unicycle for a different Frame, the field values 
> change (the methods won't, since methods are with a type).
>

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Kris De Meyer]

Thanks for the tips. There are no depreciation warnings in my own code, but 
the Sundials package which I rely on has quite a few. Although the tests 
which are really slow don't print out depreciation warnings (the printed 
warnings occur earlier), I suppose depreciation checking can still be 
slowing things down, even if no warnings are printed anymore? I guess I'll 
just have to wait to see if Sundials can catch up (I see it's failing to 
pass tests under 0.4.0). This is very annoying as I wanted to move up from 
0.3.11 because of problems in the PyCall/PyPlot/Conda packages that I don't 
seem to find a solution for in 0.3.11 but have been told are resolved in 
0.4.0.

Please accept the following not as unbridled criticism but as a way to 
improve working procedures. I've been developing Julia code since release 
v0.3.7. Unfortunately this isn't the first time that I lose many hours 
having to figure out why something that works in one release stops working 
in a subsequent one. To be honest, it's putting me off Julia and it must 
have similar effects on other potential users too. To me this points to the 
need for better procedures and guidelines in the way the language 
progresses and how the "official" packages catch up. I worked a couple of 
years for the MathWorks. Breaking backwards compatibility was generally not 
allowed, and the coordinated testing and release procedures made that 
nearly impossible. For Julia to be taken seriously by people outside the 
academic community, it would do well to start looking at how similar 
procedures can be adopted into an open-source development model. It's one 
thing to write good code and develop the "ideal" language, it's another 
thing altogether to release workable software to an outside community. 

On a slightly different note, in 2 or 3 release cycles, Matlab will have 
caught up on any performance gains Julia may have introduced (by using the 
same LLVM compiler procedures Julia uses) and then the only thing Julia 
will have going for it is that it's free. But my cost to my employers is 
such that if I lose as little as 3 days a year on compatibility issues, 
they would be better off paying for a Matlab license...

Best.

Kris





 

On Thursday, October 22, 2015 at 10:58:30 PM UTC+1, Stefan Karpinski wrote:
>
> You can try using @code_warntype to see if there are type instabilities.
>
> On Thu, Oct 22, 2015 at 5:50 PM, Gunnar Farnebäck  > wrote:
>
>> If you don't have deprecation warnings I would suspect some change in 0.4 
>> has introduced type instabilities. If you are using typed concatenations 
>> you could be hit by https://github.com/JuliaLang/julia/issues/13254.
>>
>>
>> Den torsdag 22 oktober 2015 kl. 23:03:00 UTC+2 skrev Kris De Meyer:
>>>
>>> Are there any general style guidelines for moving code from 0.3.11 to 
>>> 0.4.0? Running the unit and functionality tests for a module that I 
>>> developed under 0.3.11 in 0.4, I experience a 500 times slowdown of blocks 
>>> of code that I time with @time. 
>>>
>>> Can't even imagine where I have to start looking, and find it 
>>> flabbergasting that perfectly valid julia code under 0.3.11 (not generating 
>>> a single warning) can show such a performance degradation under 0.4.0.
>>>
>>> Anyone seen anything similar? Is there some fundamental difference in 
>>> how code is JIT-compiled under 0.4.0?
>>>
>>> Thanks,
>>>
>>> Kris
>>>
>>>
>>>
>>>
>

[julia-users] Re: good textbook for Julia software engineering?

[Sisyphuss]

On Thursday, October 22, 2015 at 6:26:29 PM UTC+2, Cedric St-Jean wrote:
>
>
> On Thursday, October 22, 2015 at 11:33:39 AM UTC-4, Sisyphuss wrote:
>>
>> I have some idea in the previous post. But it seems that they are just 
>> ignored...
>>
>
> Not to derail this thread, but FWIW, I liked your OOP equivalences, and it 
> made me consider writing shorter, focused modules.
>  
>

Thanks, Cedric, that sounds perfect. 

Re: [julia-users] Julia garbage collection - question on status and future (and interaction when using with other languages)

[Stefan Karpinski]

Yeah, I remember reading through that when he published it. Worth having
another look at...

On Fri, Oct 23, 2015 at 6:59 AM, Joshua Ballanco  wrote:

> Just to throw this out there…
>
>
> A number of years ago Mike Pall (creator and former maintainer of LuaJIT)
> outlined the beginnings of what seemed (to me at least) to be a very
> interesting variation on the tricolor GC:
> http://wiki.luajit.org/New-Garbage-Collector . Originally this was
> intended for LuaJIT v3.0, but given Mike’s recent decision to step down
> from LuaJIT maintenance, I’ve not heard what the status of further work on
> this idea is.
>
>
> For a number of reasons I think the general idea of his “arena-based,
> quad-color incremental, generational, non-copying, high-speed,
> cache-optimized garbage collector” could potentially be a good fit for
> Julia.
>
>
> On October 20, 2015 at 15:10:12, Páll Haraldsson (
> pall.haralds...@gmail.com) wrote:
>
>
> A. I know Julia had stop the world garbage collection (GC) and changed to
> generational GC in 0.4 that is faster (I've seen 10x mentioned).
>
> As far as I know, there are no knobs to turn (except possible to just to
> turn if off..), and the GC algorithm isn't selectable (except by choosing
> the older 0.3 version, but seems to be no upside to that..).
>
>
> In Go 1.5, they changed their GC (and have some impressive latency (of GC)
> numbers):
>
> "To create a garbage collector for the next decade, we turned to an
> algorithm from decades ago. Go's new garbage collector is a concurrent,
> tri-color, mark-sweep collector, an idea first proposed by Dijkstra in
> 1978. This is a deliberate divergence from most "enterprise" grade garbage
> collectors of today, and one that we believe is well suited to the
> properties of modern hardware and the latency requirements of modern
> software.
> [..]
> At a higher level, one approach to solving performance problems is to add
> GC knobs, one for each performance issue. The programmer can then turn the
> knobs in search of appropriate settings for their application. The downside
> is that after a decade with one or two new knobs each year you end up with
> the GC Knobs Turner Employment Act. Go is not going down that path. Instead
> we provide a single knob, called GOGC"
>
>
> They are not going for hard real-time GC (a hard problem.. there are hard
> real-time JVMs), it seems, but soft real-time. Just do get an overview
> picture, do we have a similar implementation? Generational, pushes down
> latency, but I think the focus in Julia is still throughput more than
> latency (or both?).
>
>
> Without being an expert on Go (or Julia) it seems the languages are
> similar enough, that we could have a GC with the same properties if we just
> wanted. But maybe the Julia community just doesn't want to, or at least as
> a priority.. Would selectable GC algorithms with different properties be
> desirable?
>
>
>
>
>
> B. A side question, I've noticed Libc.malloc etc. Say for hard (or just
> soft) real-time stuff. It seems you could use manual memory
> management/malloc/free (and would have to disable the GC I guess?). Is it
> just crazy talk/very naive that you could run Julia without the GC
> continuously (say in a game)? Or is that the intention of Libc.malloc
> access? It seems the D language allows both GC and without, is Julia just
> similar, or "not recommended in Julia"? I do not know about Go, if it
> allows both..
>
>
> C. An idea I had, and see the D guys also:
>
> http://dlang.org/garbage.html
> "Garbage collection should be implemented as a basic operating system
> kernel service. But since it is not, garbage collecting programs must carry
> around with them the garbage collection implementation."
>
> I do not really see that happening, even though memory is a global
> resource.., and ideally shouldn't be left to individual programs. Even just
> sharing a GC between say Julia and Go, I see not happening.., if you could
> get Julia and Go to work together. At best I see you could reuse Go code,
> as you can Java/JVM code, by calling it in a different process. Am I wrong?
> Strictly speaking, Python also has a GC and Julia works with Python in the
> same process. I'm not sure, but I think it may have to do with that Python
> uses reference counting (and then only full GC on top of that, is that part
> then effectively disabled by PyCall.jl?).
>
> --
> Palli.
>
>
>

[julia-users] what is best practice for allowing pre-'using' selection of a module-level setting

[Jeffrey Sarnoff]

I want to allow users of my Float12x module the choice of faster or more 
precise trig
   (at the moment, 103bits over 0..2pi is ~30% faster than 106bits; the 
more precise version also handles larger arguments).
The choice needs to be made before the using statement, as it governs the 
inclusion of either one of two small source files.

This is a binary  choice; I'd like to know how to do this for a multi-ary 
choice, too.

[julia-users] Re: are array slices views in 0.4?

[Michael Hatherly]

It’s planned for 0.5. See https://github.com/JuliaLang/julia/issues/13157 
which covers all the planned changes I think.

— Mike
​
On Friday, 23 October 2015 13:14:56 UTC+2, Neal Becker wrote:
>
> One feature I've been eager to see is that array slices become views, 
> which 
> I think was targeted for 0.4.  Is this correct?  I hate to see un-needed 
> copying. 
>
>

Re: [julia-users] Julia garbage collection - question on status and future (and interaction when using with other languages)

[Joshua Ballanco]

Just to throw this out there…

A number of years ago Mike Pall (creator and former maintainer of LuaJIT) 
outlined the beginnings of what seemed (to me at least) to be a very 
interesting variation on the tricolor GC: 
http://wiki.luajit.org/New-Garbage-Collector . Originally this was intended for 
LuaJIT v3.0, but given Mike’s recent decision to step down from LuaJIT 
maintenance, I’ve not heard what the status of further work on this idea is.

For a number of reasons I think the general idea of his “arena-based, 
quad-color incremental, generational, non-copying, high-speed, cache-optimized 
garbage collector” could potentially be a good fit for Julia.


On October 20, 2015 at 15:10:12, Páll Haraldsson (pall.haralds...@gmail.com) 
wrote:


A. I know Julia had stop the world garbage collection (GC) and changed to 
generational GC in 0.4 that is faster (I've seen 10x mentioned).

As far as I know, there are no knobs to turn (except possible to just to turn 
if off..), and the GC algorithm isn't selectable (except by choosing the older 
0.3 version, but seems to be no upside to that..).


In Go 1.5, they changed their GC (and have some impressive latency (of GC) 
numbers):

"To create a garbage collector for the next decade, we turned to an algorithm 
from decades ago. Go's new garbage collector is a concurrent, tri-color, 
mark-sweep collector, an idea first proposed by Dijkstra in 1978. This is a 
deliberate divergence from most "enterprise" grade garbage collectors of today, 
and one that we believe is well suited to the properties of modern hardware and 
the latency requirements of modern software.
[..]
At a higher level, one approach to solving performance problems is to add GC 
knobs, one for each performance issue. The programmer can then turn the knobs 
in search of appropriate settings for their application. The downside is that 
after a decade with one or two new knobs each year you end up with the GC Knobs 
Turner Employment Act. Go is not going down that path. Instead we provide a 
single knob, called GOGC"


They are not going for hard real-time GC (a hard problem.. there are hard 
real-time JVMs), it seems, but soft real-time. Just do get an overview picture, 
do we have a similar implementation? Generational, pushes down latency, but I 
think the focus in Julia is still throughput more than latency (or both?).


Without being an expert on Go (or Julia) it seems the languages are similar 
enough, that we could have a GC with the same properties if we just wanted. But 
maybe the Julia community just doesn't want to, or at least as a priority.. 
Would selectable GC algorithms with different properties be desirable?





B. A side question, I've noticed Libc.malloc etc. Say for hard (or just soft) 
real-time stuff. It seems you could use manual memory management/malloc/free 
(and would have to disable the GC I guess?). Is it just crazy talk/very naive 
that you could run Julia without the GC continuously (say in a game)? Or is 
that the intention of Libc.malloc access? It seems the D language allows both 
GC and without, is Julia just similar, or "not recommended in Julia"? I do not 
know about Go, if it allows both..


C. An idea I had, and see the D guys also:

http://dlang.org/garbage.html
"Garbage collection should be implemented as a basic operating system kernel 
service. But since it is not, garbage collecting programs must carry around 
with them the garbage collection implementation."

I do not really see that happening, even though memory is a global resource.., 
and ideally shouldn't be left to individual programs. Even just sharing a GC 
between say Julia and Go, I see not happening.., if you could get Julia and Go 
to work together. At best I see you could reuse Go code, as you can Java/JVM 
code, by calling it in a different process. Am I wrong? Strictly speaking, 
Python also has a GC and Julia works with Python in the same process. I'm not 
sure, but I think it may have to do with that Python uses reference counting 
(and then only full GC on top of that, is that part then effectively disabled 
by PyCall.jl?).

-- 
Palli.

Re: [julia-users] Re: what is best practice for allowing pre-'using' selection of a module-level setting

[Jeffrey Sarnoff]

yes -- I suppose choosing one happyesque medium is the way to go at first.

On Friday, October 23, 2015 at 12:22:24 PM UTC-4, Tom Breloff wrote:
>
> If someone does:
>
> ENV["FASTTRIG"] = "yes"
> Pkg.build("Float12x")
>
> and then in another session does Pkg.build(), (presumably for some other 
> package or update), then won't the setting be overwritten with the default 
> build setting?
>
> On Fri, Oct 23, 2015 at 12:11 PM, Jeffrey Sarnoff  > wrote:
>
>> Thank you -- clearly given.
>>
>>
>> On Friday, October 23, 2015 at 12:07:07 PM UTC-4, Steven G. Johnson wrote:
>>>
>>>
>>>
>>> On Friday, October 23, 2015 at 8:37:58 AM UTC-4, Jeffrey Sarnoff wrote:

 I want to allow users of my Float12x module the choice of faster or 
 more precise trig
(at the moment, 103bits over 0..2pi is ~30% faster than 106bits; the 
 more precise version also handles larger arguments).
 The choice needs to be made before the using statement, as it governs 
 the inclusion of either one of two small source files.

>>>
>>> It would be better to refactor your code so that the choice can be made 
>>> dynamically, after the module is loaded.
>>>
>>> However, if it is really impossible to do this dynamically, then this 
>>> kind of "static" choice would normally be made at Pkg.build time, and a 
>>> common mechanism for specifying it would be an environment variable.   The 
>>> advantage of using Pkg.build is that then your module can be precompiled.   
>>> The procedure I've used is something like:
>>>
>>> 1) create a deps/build.jl file that looks at ENV["FASTTRIG"] (or 
>>> whatever) and spits out:
>>>
>>>a) a deps/deps.jl file with a constant like `const FASTTRIG = 
>>> true`
>>>
>>>b) a deps/FASTTRIG file to save the FASTRIG setting for the next 
>>> time the file is built.  This way, if your package is rebuilt on e.g. a 
>>> Pkg.update, you won't "forget" the user's preference.i.e. you would 
>>> read ENV["FASTTRIG"] by something like:
>>>   get(ENV, "FASTTRIG", isfile("FASTTRIG") ? 
>>> readchomp("FASTTRIG") : "no")
>>>
>>> 2) in your main module file, do
>>>
>>> const depfile = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")
>>>
>>> isfile(depfile) || error("Float12x not properly installed. Please run 
>>> Pkg.build(\"Float12x\")")
>>>
>>> include(depfile) # generated by Pkg.build("Float12x")
>>>
>>>
>>> and then use the value of FASTTRIG to decide how to proceed.
>>>
>>>
>>> 3) Tell your users to do
>>>
>>>
>>> ENV["FASTTRIG"] = "yes" or "no"
>>>
>>> Pkg.build("Float12x")
>>>
>>>
>>> when they want to change the setting.
>>>
>>
>

Re: [julia-users] pop!(set, key, default)

[Stefan Karpinski]

I'd rather have it be correct than incorrect and fast so I fixed it first.
I also just don't understand the rationale for the method existing in the
first place. If you pop something from a set you should get that value
back. What use is supplying a default? If you just want to remove the value
unconditionally, you can do delete!(s, v).

On Fri, Oct 23, 2015 at 12:38 PM, Matt  wrote:

> Thanks!
> I thought it was a more efficient version than
> if key in x
>  pop!(x, key)
> end
> So yeah this method is not really useful if it is actually the same.
>
> I'm curious: isn't there some inefficiency in first testing for the key
> and then popping it out?
>
> On Friday, October 23, 2015 at 11:32:42 AM UTC-4, Stefan Karpinski wrote:
>>
>> Fixed:
>> https://github.com/JuliaLang/julia/commit/4164572c53b7c1ddb104a196d87558576031
>>
>> On Fri, Oct 23, 2015 at 10:51 AM, Stefan Karpinski 
>> wrote:
>>
>>> It's also a little unclear to me what this method is useful for.
>>>
>>> On Fri, Oct 23, 2015 at 8:53 AM, Stefan Karpinski 
>>> wrote:
>>>
 Ah, this is a bug in that pop! method. I'm working on a fix.

 On Fri, Oct 23, 2015 at 8:42 AM, Matt  wrote:

> I want to check if a key is in a set, and pop! it if it is the case.
>
>
> 1. In order to do that, I can rely on a try / catch statement (i.e.
> try pop!(x, key)) or a default key (i.e. pop!(x, key, nothing)). Is one
> preferable to the other in term of speed?
>
> 2.. pop!(x, key, nothing) returns nothing even if key is in x. Is this
> expected?
>
>
> nothing == pop!(Set(1:2), 2, nothing)
>
> true
>
>
>

>>>
>>

[julia-users] Incremental RAM usage on loop

[Diego Javier Zea]

The script 
https://github.com/diegozea/MIToS.jl/blob/master/scripts/Distances.jl is 
consuming more RAM on each iteration of *pmap*. How can I avoid it? Best

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Patrick O'Leary]

On Friday, October 23, 2015 at 10:11:16 AM UTC-5, Andreas Lobinger wrote:
>
>
> On Friday, October 23, 2015 at 3:06:13 PM UTC+2, Patrick O'Leary wrote:
>>
>> On Friday, October 23, 2015 at 6:18:43 AM UTC-5, Kris De Meyer wrote:
>>>
>>> ...and then the only thing Julia will have going for it is that it's 
>>> free. But my cost to my employers is such that if I lose as little as 3 
>>> days a year on compatibility issues, they would be better off paying for a 
>>> Matlab license... 
>>>
>>
>>
>> But if something broke in Julia, there's a chance I can fix it. Free 
>> software matters.
>>
>> Well, you can. But the average julia USER is not in the same position. 
> Fixing something in packages needs some background in development of the 
> package and fixing something in julia itself needs even more background.
>
> In any case, i prefer Open Source because there's even the possibility to 
> look inside and see how the magic is happening. 
> But the argument, 'you can help yourself' is not equal to 'you are able to 
> help yourself'.  
>

I didn't intend for the word "chance" to be read as anything other than 
"nonzero probability." 

Re: [julia-users] Re: what is best practice for allowing pre-'using' selection of a module-level setting

[Jeffrey Sarnoff]

Or have two versions that share almost all of the code.  .. appreciate the 
feedback, moving on. 

On Friday, October 23, 2015 at 1:10:22 PM UTC-4, Jeffrey Sarnoff wrote:
>
> yes -- I suppose choosing one happyesque medium is the way to go at first.
>
> On Friday, October 23, 2015 at 12:22:24 PM UTC-4, Tom Breloff wrote:
>>
>> If someone does:
>>
>> ENV["FASTTRIG"] = "yes"
>> Pkg.build("Float12x")
>>
>> and then in another session does Pkg.build(), (presumably for some other 
>> package or update), then won't the setting be overwritten with the default 
>> build setting?
>>
>> On Fri, Oct 23, 2015 at 12:11 PM, Jeffrey Sarnoff  
>> wrote:
>>
>>> Thank you -- clearly given.
>>>
>>>
>>> On Friday, October 23, 2015 at 12:07:07 PM UTC-4, Steven G. Johnson 
>>> wrote:



 On Friday, October 23, 2015 at 8:37:58 AM UTC-4, Jeffrey Sarnoff wrote:
>
> I want to allow users of my Float12x module the choice of faster or 
> more precise trig
>(at the moment, 103bits over 0..2pi is ~30% faster than 106bits; 
> the more precise version also handles larger arguments).
> The choice needs to be made before the using statement, as it governs 
> the inclusion of either one of two small source files.
>

 It would be better to refactor your code so that the choice can be made 
 dynamically, after the module is loaded.

 However, if it is really impossible to do this dynamically, then this 
 kind of "static" choice would normally be made at Pkg.build time, and a 
 common mechanism for specifying it would be an environment variable.   The 
 advantage of using Pkg.build is that then your module can be precompiled.  
  
 The procedure I've used is something like:

 1) create a deps/build.jl file that looks at ENV["FASTTRIG"] (or 
 whatever) and spits out:

a) a deps/deps.jl file with a constant like `const FASTTRIG = 
 true`

b) a deps/FASTTRIG file to save the FASTRIG setting for the next 
 time the file is built.  This way, if your package is rebuilt on e.g. a 
 Pkg.update, you won't "forget" the user's preference.i.e. you would 
 read ENV["FASTTRIG"] by something like:
   get(ENV, "FASTTRIG", isfile("FASTTRIG") ? 
 readchomp("FASTTRIG") : "no")

 2) in your main module file, do

 const depfile = joinpath(dirname(@__FILE__), "..", "deps", "deps.jl")

 isfile(depfile) || error("Float12x not properly installed. Please run 
 Pkg.build(\"Float12x\")")

 include(depfile) # generated by Pkg.build("Float12x")


 and then use the value of FASTTRIG to decide how to proceed.


 3) Tell your users to do


 ENV["FASTTRIG"] = "yes" or "no"

 Pkg.build("Float12x")


 when they want to change the setting.

>>>
>>

Re: [julia-users] Help on optimization problem

[vavasis]

Well, if this is useful, I can also point to several PhD theses on the EDM 
completion problem

  Bruce Hendrickson, Cornell, 1991

  Haw-ren Fang, Maryland, 2006

  Pratik Biswas, Stanford, 2007

  Anthony Man-Cho So, Stanford, 2007

  Nathan Krislock, Waterloo, 2010

which are all available on the web.  Probably there are other dissertations 
on the topic.  Finally, let me say sorry for misspelling Dattorro's name in 
my previous posting (the correct spelling is easier to google!)

-- Steve



On Friday, October 23, 2015 at 10:53:40 AM UTC-4, Spencer Russell wrote:
>
> Thanks for this great pointers! I’ve been browsing a couple sensor 
> localization papers (that’s the area I’m coming from) but hadn’t seen it in 
> terms of Euclidean Distance Matrix Completion. Also, Dattorro’s book looks 
> great and super relevant. I’ll keep digging into the literature.
>
> -s
>
>
> On Oct 22, 2015, at 11:10 PM, vav...@uwaterloo.ca  wrote:
>
> The problem of recovering (x,y,z) coordinates given partial pairwise 
> distance information is a famous problem in the optimization community and 
> has attracted the attention of many top people.  It is sometimes called the 
> 'Euclidean distance matrix completion' problem by mathematicians and the 
> 'sensor localization' problem by engineers.  If you search in google 
> scholar on either of these terms, you'll find many papers.  Jon Dattorio 
> wrote an entire book on the problem.
>
> -- Steve Vavasis
>
>
> On Tuesday, October 20, 2015 at 11:12:44 PM UTC-4, Spencer Russell wrote:
>>
>> I have a bunch of points in 3D whose positions I know approximately, with 
>> low-noise distance measurements between them (not necessarily fully 
>> connected). I want to improve my estimate of their 3D coordinates. This 
>> smells like an optimization problem so I’m thinking of using it as an 
>> excuse to learn JuMP. The model variables (coordinates) and objective 
>> function seem pretty straightforward (probably sum of squared distance 
>> errors for the objective?) but I don’t know enough about the various 
>> solvers to know which one to use. I know a tiny bit about optimization 
>> (I’ve implemented simplex, L-M, and conjugate GD for a class) but get 
>> quickly outside my depth when it gets theoretical. 
>>
>> Without any known positions the system is underconstrained because the 
>> whole collection could can translate, rotate, and reflect, but if I set 3 
>> non-colinear points at known positions it should be solvable. I can also 
>> supply pretty good guesses for initial values on locations, at least close 
>> enough that the closest minima is the one I want (I think). 
>>
>> so in short my questions boil down to: 
>>
>> 1. What solver is appropriate for this sort of problem? (or should I just 
>> go with a spring-force algorithm like in GraphLayout.jl?) 
>> 2. (JuMP Specific) - Should I specify my known positions as model 
>> variables with equality constraints, or just normal julia variables that 
>> show up in my objective function? 
>>
>> Thanks! 
>>
>> -s
>
>
>

Re: [julia-users] pop!(set, key, default)

[Matt]

Thanks! 
I thought it was a more efficient version than 
if key in x
 pop!(x, key)
end
So yeah this method is not really useful if it is actually the same.

I'm curious: isn't there some inefficiency in first testing for the key and 
then popping it out?

On Friday, October 23, 2015 at 11:32:42 AM UTC-4, Stefan Karpinski wrote:
>
> Fixed: 
> https://github.com/JuliaLang/julia/commit/4164572c53b7c1ddb104a196d87558576031
>
> On Fri, Oct 23, 2015 at 10:51 AM, Stefan Karpinski  > wrote:
>
>> It's also a little unclear to me what this method is useful for.
>>
>> On Fri, Oct 23, 2015 at 8:53 AM, Stefan Karpinski > > wrote:
>>
>>> Ah, this is a bug in that pop! method. I'm working on a fix.
>>>
>>> On Fri, Oct 23, 2015 at 8:42 AM, Matt  
>>> wrote:
>>>
 I want to check if a key is in a set, and pop! it if it is the case.


 1. In order to do that, I can rely on a try / catch statement (i.e. 
 try pop!(x, key)) or a default key (i.e. pop!(x, key, nothing)). Is one 
 preferable to the other in term of speed?

 2.. pop!(x, key, nothing) returns nothing even if key is in x. Is this 
 expected?


 nothing == pop!(Set(1:2), 2, nothing)

 true



>>>
>>
>

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Kris De Meyer]

Thanks for all the information. 

The problem is probably not with the language releases itself (I take your 
word for it that point releases don't introduce incompatibilities) but with 
the package releases. Sometimes they lag behind  the language release (as 
I'm currently experiencing with Sundials) and sometimes they may well pass 
individual tests but then fail on real-world package dependencies which 
aren't noticed because they are not covered by tests. To give an example of 
the latter, a few weeks ago, when I started to test a major code re-write 
of my own, I got a set of "convert" errors. After digging around 
fruitlessly in my own code for several hours (the obscurity of the message 
didn't help), I finally discovered it was an error triggered by loading one 
package before another (forgot now which one it was, I think Colors.jl was 
involved). A third problem is that by depending on Python, some packages 
introduce all of the nightmares (and then some) of the multiverse of Python 
releases, especially on Linux (which is the cause of the error I'm 
experiencing with a 0.3.11 Julia/IJulia/Conda installation on Linux at the 
moment). A fourth problem is that, despite the assertion that I've seen 
that for-loops over Arrays should run at the same speed as vectorized code, 
I've found on several occasions that for loops are still quite a bit 
slower. It may be that I'm not writing it in the most optimal way but as 
far as I know I am following the documentation and guidelines that exist.

I know that all of these problems are solvable, but that's not the point I 
am trying to make. All of these issues take time that keep me away from 
writing the software that I am hired to write in a limited number of days, 
and my employers may not be interested in the reasons for why I can't get 
it to work on time. Please also note that when in the midst of a crisis and 
up against deadlines, I (and probably other users in similar situations) 
may not have time to file bug reports, and not even have the time to note 
down how I fixed or circumvented a certain problem.

I understand that my criticism may sound misplaced because I know that 
Julia is in early development, but as a "real" user of Julia (not involved 
in the development itself, but attempting to write real production code, 
rather than using it as an academic exercise), I'd rather let you know of 
my experience of using it. 

Thanks! And keep up the good work. Free software matters indeed - though so 
does paid-for.







  


On Friday, October 23, 2015 at 1:43:35 PM UTC+1, Stefan Karpinski wrote:
>
> Yes, my experience with upgrading large code bases to new point releases 
> is that no changes are required. Kris, if your experience with point 
> releases has been otherwise, it would be good to hear about specific issues 
> since those releases are strictly bug fix releases. Perhaps you are talking 
> about the last major upgrade from 0.2.x. to 0.3.x?
>
> On Fri, Oct 23, 2015 at 8:25 AM, Tony Kelman  > wrote:
>
>> >  I've been developing Julia code since release v0.3.7. Unfortunately 
>> this isn't the first time that I lose many hours having to figure out why 
>> something that works in one release stops working in a subsequent one.
>>
>> This surprises me. 0.4.0 is the only release that was intended to be 
>> breaking in any meaningful way since 0.3.0. Every point release 
>> announcement has said "We would like to get feedback if someone has a 
>> working program that breaks after this upgrade." I meant that seriously, if 
>> an unintended breaking change was made in any of the 0.3.x series and an 
>> issue was reported about it, we would find a way to fix it (for example, 
>> there was a problem in the embedding API that led us to put out 0.3.5 on an 
>> accelerated schedule to fix it). The same holds for 0.4.x relative to 0.4.0.
>>
>
>

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Christoph Ortner]

Apparently yes. For me it is very counterintuitive that this would be the 
default behaviour. But presumably there was a lot of discussion that this 
is desirable.

(a) What are reasons, other than performance?

(b) Is this still under discussion or pretty much settled?

(c) if I want to write code now that shouldn't break with 0.5, what should 
I do?

Thanks,
  Christoph


On Friday, 23 October 2015 20:04:58 UTC+1, Christoph Ortner wrote:
>
>
> If I write into a view, does it change the original array? 
>
> Christoph
>

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Stefan Karpinski]

Yes, that's the whole premise of a view.

On Fri, Oct 23, 2015 at 3:04 PM, Christoph Ortner <
christophortn...@gmail.com> wrote:

>
> If I write into a view, does it change the original array?
>
> Christoph
>

Re: [julia-users] Julia and Object-Oriented Programming

[ssarkarayushnetdev]

Inheritance should be understood in terms of properties (and operations on 
properties) of a classes and sub-classes.  In functional programming 
language,
application of functions like *func_1*(x, y, *func_2*(z, p)) is implicitly 
exploiting the idea of inheritance. Making such relationships explicit
in programming languages will lead to much less errors at run time with 
compile time ( definition and declaration ) checks.


On Friday, October 23, 2015 at 7:59:26 AM UTC-7, Brendan Tracey wrote:
>
> On Friday, October 23, 2015 at 7:38:37 AM UTC-6, Abe Schneider wrote:
>>
>> An OO approach is really just specifying an interface in a formal manner. 
>> The second you write any type of interface, you always risk making a choice 
>> that will haunt you down the road. I don't see the difference between
>>
>> class Foo {
>>   float getX() { ... }
>>   float getY() { ... }
>> }
>>
>> and:
>>
>> type Foo { ... }
>> function getX(f::Foo) -> float { ... }
>> function getY(f::Foo) -> float { ... }
>>
>> except that an instance of `foo` is being passed implicitly in the first 
>> case. To that extent, I would say Julia OO (i.e. on the dispatching). Where 
>> it is not OO is in the layout of the data.
>>
>
> In some  languages there is no difference. In Go, a method is exactly a 
> function that can be called with a special syntax. One can do
>
> c := mat64.NewDense(5, 5, nil)
> c.Mul(a, b)
>  
> or one can do
> (*mat64.Dense).Mul(c, a, b)
>
> There aren't many cases where one would actually want to do the latter, 
> but it makes the definition of behavior clear.
>
> I think the difference in Julia would be the interaction with multiple 
> dispatch. In the former case, it seems like getX would live in its own 
> namespace (only accessible through the Foo class), while in the latter case 
> it would add to the many definitions of getX.
>
>
> It is true that with a more OO language, because of its formalism, you run 
>> the risk of declaring variable types in a parent class that might not be 
>> correct in the child classes, so some planning is required. However, what 
>> you gain is better code in the long run. For example, I might have:
>>
>> abstract Baz
>>
>> type Foo <: Baz {
>>   x::Float
>> }
>>
>> type Bar <: Baz {
>>   x::Float
>> }
>>
>> and an interface:
>>
>> getX{T <: Baz}(b::T) -> Float = b.x
>>
>>
>> which works fine, except maybe someone else comes along and writes:
>>
>> type Blob <: Baz {
>>   myX::Float
>> }
>>
>> Now to fix your interface, you have to write a separate `getX` for 
>> `Blob`. This might not seem like a big deal, except you might not catch 
>> this issue until run time (I don't think there is a way a static-checker 
>> could identify the problem). Imagine a large library or base of code, with 
>> many people working on the code, and you suddenly are exposed to a large 
>> number of issues.
>>
>
> This seems more like an issue of compiler vs. no compiler issue than one 
> with OO vs. multiple dispatch.
>  
> You can do OO without inheritance (or OO like things if your definition of 
> OO includes inheritance). In Go, one would define 
>
> type GetXer interface {
>  GetX() float64
> }
>
> I could then write a function
> func SquareX(g GetXer) float64 {
>  v := g.GetX()
>  return v * v
> }
>
> Now, any type that has a GetX method can be passed to SquareX. There is no 
> inheritance necessary to make this happen.
>
>

Re: [julia-users] Julia and Object-Oriented Programming

[Brendan Tracey]

On Friday, October 23, 2015 at 7:22:28 AM UTC-6, Abe Schneider wrote:
>
> Ah, okay, that is different then. What's the advantage of creating a new 
> method versus copying the fields? I would imagine there is a penalty with 
> each deference you have to follow in order to make that work.
>

The advantage is simplicity. A unicycle is a Frame, a Seat, and a Wheel, no 
more, no less (unless we added more to the definition of course).  
Inheritance gets complicated fast in C++ (again, can't speak for Scala) 
with virtual functions, virtual vs. nonvirtual descructors. Additionally, 
Unicycle doesn't inherit from anything. It's not a subclass of anything. It 
can be used however. 

As to dereferencing, types in Go are not boxed like in Java (for example). 
We could construct Unicycle as 
type Unicycle struct {
 Frame
 Seat
 Wheel
}

or as 
type Unicycle struct {
 *Frame
 *Seat
 *Wheel
}

or any combination you'd like. In the latter case they are pointers, and in 
the former they are not. So, first of all you can avoid the dereferencing 
by not having pointers. Secondly, in Go all of the types are known at 
compile type, so I believe you can call the function without going through 
extra deferencing, even in the case where they are pointers.

Re: [julia-users] Help on optimization problem

[Spencer Russell]

Thanks for this great pointers! I’ve been browsing a couple sensor localization 
papers (that’s the area I’m coming from) but hadn’t seen it in terms of 
Euclidean Distance Matrix Completion. Also, Dattorro’s book looks great and 
super relevant. I’ll keep digging into the literature.

-s


> On Oct 22, 2015, at 11:10 PM, vava...@uwaterloo.ca wrote:
> 
> The problem of recovering (x,y,z) coordinates given partial pairwise distance 
> information is a famous problem in the optimization community and has 
> attracted the attention of many top people.  It is sometimes called the 
> 'Euclidean distance matrix completion' problem by mathematicians and the 
> 'sensor localization' problem by engineers.  If you search in google scholar 
> on either of these terms, you'll find many papers.  Jon Dattorio wrote an 
> entire book on the problem.
> 
> -- Steve Vavasis
> 
> 
> On Tuesday, October 20, 2015 at 11:12:44 PM UTC-4, Spencer Russell wrote:
> I have a bunch of points in 3D whose positions I know approximately, with 
> low-noise distance measurements between them (not necessarily fully 
> connected). I want to improve my estimate of their 3D coordinates. This 
> smells like an optimization problem so I’m thinking of using it as an excuse 
> to learn JuMP. The model variables (coordinates) and objective function seem 
> pretty straightforward (probably sum of squared distance errors for the 
> objective?) but I don’t know enough about the various solvers to know which 
> one to use. I know a tiny bit about optimization (I’ve implemented simplex, 
> L-M, and conjugate GD for a class) but get quickly outside my depth when it 
> gets theoretical. 
> 
> Without any known positions the system is underconstrained because the whole 
> collection could can translate, rotate, and reflect, but if I set 3 
> non-colinear points at known positions it should be solvable. I can also 
> supply pretty good guesses for initial values on locations, at least close 
> enough that the closest minima is the one I want (I think). 
> 
> so in short my questions boil down to: 
> 
> 1. What solver is appropriate for this sort of problem? (or should I just go 
> with a spring-force algorithm like in GraphLayout.jl?) 
> 2. (JuMP Specific) - Should I specify my known positions as model variables 
> with equality constraints, or just normal julia variables that show up in my 
> objective function? 
> 
> Thanks! 
> 
> -s

Re: [julia-users] Re: are array slices views in 0.4?

[Tim Holy]

This will be a ridiculous charicature, but this issue has come up so many 
times that I think a little charicaturization is timely.

With regards to arrays and views, the "big change" in julia 0.5 will 
essentially be a 1-line change:

 getindex(A, indexes...) = slice(A, indexes...)

Since slice is already in base, you can use it today :-).

There's much more to it than that: SubArrays don't always check bounds, so 
currently this is At Your Own Risk. There are some difficulties in reshaping 
SubArrays.

But if the fundamental question is, "can I have performant and very flexible 
views?" the answer is: you already have them.

Best,
--Tim

On Friday, October 23, 2015 04:36:03 AM Michael Hatherly wrote:
> It’s planned for 0.5. See https://github.com/JuliaLang/julia/issues/13157
> which covers all the planned changes I think.
> 
> — Mike
> ​
> 
> On Friday, 23 October 2015 13:14:56 UTC+2, Neal Becker wrote:
> > One feature I've been eager to see is that array slices become views,
> > which
> > I think was targeted for 0.4.  Is this correct?  I hate to see un-needed
> > copying.

Re: [julia-users] Re: map() vs broadcast()

[Ján Dolinský]

Thanks for the update. I read the link. I did some further investigation 
e.g.

N = 5_000_000
A = rand(N)

R = UnitRange[ 1:round(Int, a*10) for a in A]

function testMap1(R)

  tmp = zeros(Int, length(R))
  for i in eachindex(R)
tmp[i] = length(R[i])
  end

  sum(tmp)
end

function testMap2(R)
  sum(map(length, R))
end

testMap1(R)
testMap2(R)

@time testMap1(R)
0.572483 seconds (7 allocations: 38.147 MB, 0.23% gc time)
24992990
@time testMap2(R)
  0.279889 seconds (8 allocations: 38.147 MB, 0.62% gc time)
24992990

I wonder why here is map() so efficient, even faster than de-vectorized 
loop version in testMap1().

Regards,
Jan

Dňa piatok, 23. októbra 2015 11:12:06 UTC+2 Kristoffer Carlsson napísal(-a):
>
> This is not a new issue.
>
> You are simply bumping into the problem that passing functions as 
> arguments incur a cost every time the function is called.
>
> If you want to compare it with map! in base you should do the following:
>
> function mape4a(f, A, F)
>
> tmp = similar(A)
> for i in eachindex(A)
> tmp[i] = f(A[i], F[i])
> end
>
> 100 * sumabs(tmp) / length(A)
> end
>
>
> @time mape4a(_f A,F)
>  0.348988 seconds (20.00 M allocations: 343.323 MB, 8.25% gc time)
>
> There are plans on fixing this, see 
> https://github.com/JuliaLang/julia/pull/13412
>
> On Friday, October 23, 2015 at 10:58:06 AM UTC+2, Ján Dolinský wrote:
>>
>> versioninfo()
>> Julia Version 0.4.0
>> Commit 0ff703b* (2015-10-08 06:20 UTC)
>> Platform Info:
>>   System: Linux (x86_64-linux-gnu)
>>   CPU: Intel(R) Core(TM) i5-4300U CPU @ 1.90GHz
>>   WORD_SIZE: 64
>>   BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY 
>> Haswell)
>>   LAPACK: liblapack.so.3
>>   LIBM: libopenlibm
>>   LLVM: libLLVM-3.3
>>
>> Hi Milan,
>>
>> The above is the versioninfo() output. I am exploring this further, using 
>> map() instead of map!() give me 3 time 5 million allocations as opposed to 
>> map!() with 4 times 5 million allocations. The "for" cycle in either map or 
>> map!() should not allocate that much memory.  See my devectorized example 
>> in the previous post.
>>
>> Shall I file an issue, please advise me on how to do it. In general, I 
>> think map() and broadcast() should have about the same performance in the 
>> example given in the beginning of this thread.
>>
>> Thanks,
>> Jan
>>
>> Dňa piatok, 23. októbra 2015 10:44:01 UTC+2 Milan Bouchet-Valat 
>> napísal(-a):
>>>
>>> This sounds suspicious to me. If you can file an issue with a 
>>> reproducible example, you'll soon get feedback about what's going on 
>>> here. 
>>>
>>> Please report the output of versioninfo() there too. I assume this is 
>>> on 0.4? 
>>>
>>>
>>> Regards 
>>>
>>> Le vendredi 23 octobre 2015 à 00:42 -0700, Ján Dolinský a écrit : 
>>> > ## 2 argument 
>>> > function map!{F}(f::F, dest::AbstractArray, A::AbstractArray, 
>>> > B::AbstractArray) 
>>> > for i = 1:length(A) 
>>> > dest[i] = f(A[i], B[i]) 
>>> > end 
>>> > return dest 
>>> > end 
>>> > 
>>> > The above is the map!() implementation in abstractarray.jl. Should it 
>>> > return "dest" if it is an in-place function ? Is there any 
>>> > fundamental difference between my mape4a() and map!() in 
>>> > abstractarray.jl ? 
>>> > 
>>> > Thanks, 
>>> > Jan 
>>> > 
>>> > Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(-a): 
>>> > > Hi Glen, 
>>> > > 
>>> > > Thanks for the investigation. I am afraid the for loop in map!() is 
>>> > > not the source of the issue. Consider the folowing: 
>>> > > 
>>> > > _f(a,f) = (a - f) / a 
>>> > > 
>>> > > function mape4(A, F) 
>>> > > # A - actual target values 
>>> > > # F - forecasts (model estimations) 
>>> > > 
>>> > >   tmp = similar(A) 
>>> > >   map!(_f, tmp, A, F) 
>>> > >   100 * sumabs(tmp) / length(A) 
>>> > > 
>>> > > end 
>>> > > 
>>> > > function mape4a(A, F) 
>>> > > 
>>> > > tmp = similar(A) 
>>> > > for i in eachindex(A) 
>>> > > tmp[i] = _f(A[i], F[i]) 
>>> > > end 
>>> > > 100 * sumabs(tmp) / length(A) 
>>> > > end 
>>> > > 
>>> > > @time mape4(A,F) 
>>> > >   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time) 
>>> > > 832.852597807525 
>>> > > 
>>> > > @time mape4a(A,F) 
>>> > >   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time) 
>>> > > 832.852597807525 
>>> > > 
>>> > > The for loop in mape4a() does not do 4 * 5 milion allocations, 
>>> > > neither should do the loop in map!(). Is this possibly a bug ? 
>>> > > 
>>> > > Thanks, 
>>> > > Jan 
>>> > > 
>>> > > Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a): 
>>> > > > I'm uncertain, but I think I may have figured out what's going 
>>> > > > on. 
>>> > > > 
>>> > > > The hint lies in the number of allocations - map! has 20 million 
>>> > > > allocations, while broadcast! has just 5. So I had a look at how 
>>> > > > the two functions are implemented. 
>>> > > > 
>>> > > > map! is implemented in perhaps the simplest way you can think of 
>>> > > > - for 

[julia-users] Re: Julia garbage collection - question on status and future (and interaction when using with other languages)

[Jonathan Malmaud]

On Tuesday, October 20, 2015 at 8:10:07 AM UTC-4, Páll Haraldsson wrote:
>
>
> A. I know Julia had stop the world garbage collection (GC) and changed to 
> generational GC in 0.4 that is faster (I've seen 10x mentioned).
>
> As far as I know, there are no knobs to turn (except possible to just to 
> turn if off..), and the GC algorithm isn't selectable (except by choosing 
> the older 0.3 version, but seems to be no upside to that..).
>
>
> In Go 1.5, they changed their GC (and have some impressive latency (of GC) 
> numbers):
>
> "To create a garbage collector for the next decade, we turned to an 
> algorithm from decades ago. Go's new garbage collector is a concurrent, 
> tri-color, mark-sweep collector, an idea first proposed by Dijkstra in 
> 1978. This is a deliberate divergence from most "enterprise" grade garbage 
> collectors of today, and one that we believe is well suited to the 
> properties of modern hardware and the latency requirements of modern 
> software.
> [..]
> At a higher level, one approach to solving performance problems is to add 
> GC knobs, one for each performance issue. The programmer can then turn the 
> knobs in search of appropriate settings for their application. The downside 
> is that after a decade with one or two new knobs each year you end up with 
> the GC Knobs Turner Employment Act. Go is not going down that path. Instead 
> we provide a single knob, called GOGC"
>
>
> They are not going for hard real-time GC (a hard problem.. there are hard 
> real-time JVMs), it seems, but soft real-time. Just do get an overview 
> picture, do we have a similar implementation? Generational, pushes down 
> latency, but I think the focus in Julia is still throughput more than 
> latency (or both?).
>
>
> Without being an expert on Go (or Julia) it seems the languages are 
> similar enough, that we could have a GC with the same properties if we just 
> wanted. But maybe the Julia community just doesn't want to, or at least as 
> a priority.. Would selectable GC algorithms with different properties be 
> desirable?
>
>
> I don't think the throughput of Go's GC is all that much better than 
better than Julia's - it's really its latency that is much better. But 
latency is fairly irrelevant for large batch jobs, which I suspect is what 
most people using Julia are concerned with.  

>
>
>
> B. A side question, I've noticed Libc.malloc etc. Say for hard (or just 
> soft) real-time stuff. It seems you could use manual memory 
> management/malloc/free (and would have to disable the GC I guess?). Is it 
> just crazy talk/very naive that you could run Julia without the GC 
> continuously (say in a game)? Or is that the intention of Libc.malloc 
> access? It seems the D language allows both GC and without, is Julia just 
> similar, or "not recommended in Julia"? I do not know about Go, if it 
> allows both..
>
>
>
You can use Libc.malloc if you want and it will work for fine: 
x=pointer_to_array(convert(Ptr{Float64}, Libc.malloc(10sizeof(Float64))), 
10, false)
x[4]=5.2
...

You don't have disable the GC For this to work: the last parameter to 
'pointer_to_array' indicates to the GC to not touch this memory.




 

> C. An idea I had, and see the D guys also:
>
> http://dlang.org/garbage.html
> "Garbage collection should be implemented as a basic operating system 
> kernel service. But since it is not, garbage collecting programs must carry 
> around with them the garbage collection implementation."
>
> I do not really see that happening, even though memory is a global 
> resource.., and ideally shouldn't be left to individual programs. Even just 
> sharing a GC between say Julia and Go, I see not happening.., if you could 
> get Julia and Go to work together. At best I see you could reuse Go code, 
> as you can Java/JVM code, by calling it in a different process. Am I wrong? 
> Strictly speaking, Python also has a GC and Julia works with Python in the 
> same process. I'm not sure, but I think it may have to do with that Python 
> uses reference counting (and then only full GC on top of that, is that part 
> then effectively disabled by PyCall.jl?).
>

Python is indeed reference-incremented. PyCall.jl manually increments and 
decrements the reference count of Python objects as they're created and 
freed.

Nothing in general though stops two different GCs from two different 
libraries (eg, libgo and libjulia) from running in the same process, each 
responsible for its own objects.

>
> -- 
> Palli.
>
>
>

Re: [julia-users] Re: Julia garbage collection - question on status and future (and interaction when using with other languages)

[Yichao Yu]

On Fri, Oct 23, 2015 at 10:40 AM, Jonathan Malmaud  wrote:
>
>
>
> On Tuesday, October 20, 2015 at 8:10:07 AM UTC-4, Páll Haraldsson wrote:
>>
>>
>> A. I know Julia had stop the world garbage collection (GC) and changed to
>> generational GC in 0.4 that is faster (I've seen 10x mentioned).
>>
>> As far as I know, there are no knobs to turn (except possible to just to
>> turn if off..), and the GC algorithm isn't selectable (except by choosing
>> the older 0.3 version, but seems to be no upside to that..).
>>
>>
>> In Go 1.5, they changed their GC (and have some impressive latency (of GC)
>> numbers):
>>
>> "To create a garbage collector for the next decade, we turned to an
>> algorithm from decades ago. Go's new garbage collector is a concurrent,
>> tri-color, mark-sweep collector, an idea first proposed by Dijkstra in 1978.
>> This is a deliberate divergence from most "enterprise" grade garbage
>> collectors of today, and one that we believe is well suited to the
>> properties of modern hardware and the latency requirements of modern
>> software.
>> [..]
>> At a higher level, one approach to solving performance problems is to add
>> GC knobs, one for each performance issue. The programmer can then turn the
>> knobs in search of appropriate settings for their application. The downside
>> is that after a decade with one or two new knobs each year you end up with
>> the GC Knobs Turner Employment Act. Go is not going down that path. Instead
>> we provide a single knob, called GOGC"
>>
>>
>> They are not going for hard real-time GC (a hard problem.. there are hard
>> real-time JVMs), it seems, but soft real-time. Just do get an overview
>> picture, do we have a similar implementation? Generational, pushes down
>> latency, but I think the focus in Julia is still throughput more than
>> latency (or both?).
>>
>>
>> Without being an expert on Go (or Julia) it seems the languages are
>> similar enough, that we could have a GC with the same properties if we just
>> wanted. But maybe the Julia community just doesn't want to, or at least as a
>> priority.. Would selectable GC algorithms with different properties be
>> desirable?
>>
>>
> I don't think the throughput of Go's GC is all that much better than better
> than Julia's - it's really its latency that is much better. But latency is
> fairly irrelevant for large batch jobs, which I suspect is what most people
> using Julia are concerned with.
>>
>>
>>
>>
>> B. A side question, I've noticed Libc.malloc etc. Say for hard (or just
>> soft) real-time stuff. It seems you could use manual memory
>> management/malloc/free (and would have to disable the GC I guess?). Is it
>> just crazy talk/very naive that you could run Julia without the GC
>> continuously (say in a game)? Or is that the intention of Libc.malloc
>> access? It seems the D language allows both GC and without, is Julia just
>> similar, or "not recommended in Julia"? I do not know about Go, if it allows
>> both..
>>
>>
>
> You can use Libc.malloc if you want and it will work for fine:
> x=pointer_to_array(convert(Ptr{Float64}, Libc.malloc(10sizeof(Float64))),
> 10, false)
> x[4]=5.2
> ...
>
> You don't have disable the GC For this to work: the last parameter to
> 'pointer_to_array' indicates to the GC to not touch this memory.
>

You probably want to pass `true`, which will let the GC call `free` on
it automatically.

If you want to manually manage the pointer, you can pass `false`. Note
that this will
still allocate the `Array` object and pass `false` will probably not
make the GC run
less frequently.

Also note that `malloc/free` are not hard real time either, you
basically cannot have
any sort of memory allocation for that.

>
>
>
>
>>
>> C. An idea I had, and see the D guys also:
>>
>> http://dlang.org/garbage.html
>> "Garbage collection should be implemented as a basic operating system
>> kernel service. But since it is not, garbage collecting programs must carry
>> around with them the garbage collection implementation."
>>
>> I do not really see that happening, even though memory is a global
>> resource.., and ideally shouldn't be left to individual programs. Even just
>> sharing a GC between say Julia and Go, I see not happening.., if you could
>> get Julia and Go to work together. At best I see you could reuse Go code, as
>> you can Java/JVM code, by calling it in a different process. Am I wrong?
>> Strictly speaking, Python also has a GC and Julia works with Python in the
>> same process. I'm not sure, but I think it may have to do with that Python
>> uses reference counting (and then only full GC on top of that, is that part
>> then effectively disabled by PyCall.jl?).
>
>
> Python is indeed reference-incremented. PyCall.jl manually increments and
> decrements the reference count of Python objects as they're created and
> freed.
>
> Nothing in general though stops two different GCs from two different
> libraries (eg, libgo and libjulia) from running in the same process, each

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Andreas Lobinger]

On Friday, October 23, 2015 at 3:06:13 PM UTC+2, Patrick O'Leary wrote:
>
> On Friday, October 23, 2015 at 6:18:43 AM UTC-5, Kris De Meyer wrote:
>>
>> ...and then the only thing Julia will have going for it is that it's 
>> free. But my cost to my employers is such that if I lose as little as 3 
>> days a year on compatibility issues, they would be better off paying for a 
>> Matlab license... 
>>
>
>
> But if something broke in Julia, there's a chance I can fix it. Free 
> software matters.
>
> Well, you can. But the average julia USER is not in the same position. 
Fixing something in packages needs some background in development of the 
package and fixing something in julia itself needs even more background.

In any case, i prefer Open Source because there's even the possibility to 
look inside and see how the magic is happening. 
But the argument, 'you can help yourself' is not equal to 'you are able to 
help yourself'.  

[julia-users] Re: map() vs broadcast()

[Ján Dolinský]

Yes, indeed, mape4b() is the most efficient because reduction is included 
in the loop. This is actually my mape5(). I am preparing this for my 
colleagues as an introductory course to Julia :).

Thanks,
Jan

Dňa piatok, 23. októbra 2015 11:52:30 UTC+2 DNF napísal(-a):
>
> I guess this is mostly about the behaviour of map!, but for avoiding 
> allocations I think this makes more sense:
>
> function mape4b(A, F)
> tmp = zero(eltype(A))
> N = length(A)
> for i in 1:N
> tmp += abs(_f(A[i], F[i]))
> end
> return 100 * tmp / N
> end
>
> For some reason, the time macro doesn't give me any number for allocation 
> in mape4b, but it's 20-25% faster than mape4a and almost twice as fast as 
> the broadcast! version on my computer.
>
> I tried using @inbounds but that had no effect whatsoever.
>

[julia-users] Re: map() vs broadcast()

[Ján Dolinský]

Hi Glen,

Thanks for the investigation. I am afraid the for loop in map!() is not the 
source of the issue. Consider the folowing:

_f(a,f) = (a - f) / a

function mape4(A, F)
# A - actual target values
# F - forecasts (model estimations)

  tmp = similar(A)
  map!(_f, tmp, A, F)
  100 * sumabs(tmp) / length(A)

end

function mape4a(A, F)

tmp = similar(A)
for i in eachindex(A)
tmp[i] = _f(A[i], F[i])
end
100 * sumabs(tmp) / length(A)
end

@time mape4(A,F)
  0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time)
832.852597807525

@time mape4a(A,F)
  0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time)
832.852597807525

The for loop in mape4a() does not do 4 * 5 milion allocations, neither 
should do the loop in map!(). Is this possibly a bug ?

Thanks,
Jan

Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a):
>
> I'm uncertain, but I think I may have figured out what's going on.
>
> The hint lies in the number of allocations - map! has 20 million 
> allocations, while broadcast! has just 5. So I had a look at how the two 
> functions are implemented.
>
> map! is implemented in perhaps the simplest way you can think of - for 
> i=1:length(A) dest[i]=f(A[i],B[i]); end - which means that it has to store 
> four values per iteration - i, A[i], B[i], and f(A[i],B[i]). Thus, 4 times 
> 5 million allocations.
>
> broadcast! is using a cache to store values, instead, and I believe it's 
> generating instructions using a macro instead of a regular loop, thus 
> avoiding the assignments for i. As such, it doesn't need to store anything 
> except for the initial caches, and after that it just overwrites the 
> existing values. Unfortunately, that's as much as I can figure out from 
> broadcast!, because it uses a lot of macros and a lot of relatively opaque 
> structure.
>
> I'm also not entirely sure how it avoids the assignments necessary in the 
> function call.
>
> On Friday, 23 October 2015 01:54:14 UTC+10, Ján Dolinský wrote:
>>
>> Hi,
>>
>> I am exploring Julia's map() and broadcast() functions. I did a simple 
>> implementation of MAPE (mean absolute percentage error) using broadcast() 
>> and map(). Interestingly, the difference in performance was huge.
>>
>> A = rand(5_000_000)
>> F = rand(5_000_000)
>>
>> _f(a,f) = (a - f) / a
>>
>> function mape3(A, F)
>> # A - actual target values
>> # F - forecasts (model estimations)
>>
>>   tmp = similar(A)
>>   broadcast!(_f, tmp, A, F)
>>   100 * sumabs(tmp) / length(A)
>>
>> end
>>
>> function mape4(A, F)
>> # A - actual target values
>> # F - forecasts (model estimations)
>>
>>   tmp = similar(A)
>>   map!(_f, tmp, A, F)
>>   100 * sumabs(tmp) / length(A)
>>
>> end
>>
>> @time mape3(A,F) # after JIT warm-up
>>   0.038686 seconds (8 allocations: 38.147 MB, 2.25% gc time)
>> 876.4813057521973
>>
>> @time mape4(A,F) # after JIT warm-up
>>   0.457771 seconds (20.00 M allocations: 343.323 MB, 11.29% gc time)
>> 876.4813057521973
>>
>> I wonder why map() is so much slower ?
>>
>> Thanks,
>> Jan
>>
>

Re: [julia-users] C-style casts

[Yichao Yu]

(a % UInt32) where a is a UInt64

On Fri, Oct 23, 2015 at 5:12 PM, Damien  wrote:
> How do I cast a UInt64 to a UInt32, discarding the extra bits, and with no
> error checking (so it can work inside @simd loops)?
>
> There are algorithms that require this, eg.
> http://www.pcg-random.org/download.html

[julia-users] Re: Building a Binomial Tree

[Eric Forgy]

Hi Christopher,

I am just learning then Julian way myself, but one thing that "might" be 
better than an array of growing arrays is to note that a binary tree can be 
laid into a matrix, i.e. Array{Float64,2}, by rotating it. This is nice in 
case you ever need a ternary tree, which could be represented by 
Array{Float64,3}, etc.

Another thing you might consider is to treat the tree as a directed graph 
and look at incorporating one of the Julia graph theory packages.

On the topic, I have a paper (shameless plug alert!) you might be 
interested in:

   - *Financial Modelling Using Discrete Stochastic Calculus* 
   
   
More papers here .


On Saturday, October 24, 2015 at 7:50:59 AM UTC+8, Christopher Alexander 
wrote:
>
> Hello all,
>
> I am trying to write a method that builds a binomial tree for option 
> pricing.  I am trying to set up my tree like this:
>
> function build_tree(s_opt::StockOption)
>   u = 1 + s_opt.pu
>   d = 1 - s_opt.pd
>   # qu = (exp((s_opt.r - s_opt.div) * s_opt.dt) - d) / (u - d)
>   # qd = 1 - qu
>
>   # init array
>stockTree = Vector{Float64}[ zeros(m) for m = 1:s_opt.N + 1]
>
>   for i = 1:s_opt.N + 1
> for j = 1:i
>   stockTree[i][j] = s_opt.S0 * u ^ (j-1) * d ^ (i - j)
> end
>   end
>
>   return stockTree
> end
>
> Is this the most "Julian" way to do this (I didn't find really any modules 
> that would suit this purpose)?
>
> Here is what prints out:
>
> *3-element Array{Array{Float64,1},1}:*
>
> * [50.0]   *
>
> * [40.0,60.0]  *
>
> * [32.01,48.0,72.0]*
>
> I suppose also I could default the [1][1] state to the spot price (S0), so 
> then I don't need to alter the looping range and instead start the range at 
> 2.
>
> Thanks!
>
> Chris
>

[julia-users] Building a Bionmial Tree

[Christopher Alexander]

Hello all,

I am trying to write a method that builds a binomial tree for option 
pricing.  I am trying to set up my tree like this:

function build_tree(s_opt::StockOption)
  u = 1 + s_opt.pu
  d = 1 - s_opt.pd
  # qu = (exp((s_opt.r - s_opt.div) * s_opt.dt) - d) / (u - d)
  # qd = 1 - qu

  # init array
   stockTree = Vector{Float64}[ zeros(m) for m = 1:s_opt.N + 1]

  for i = 1:s_opt.N + 1
for j = 1:i
  stockTree[i][j] = s_opt.S0 * u ^ (j-1) * d ^ (i - j)
end
  end

  return stockTree
end

Is this the most "Julian" way to do this (I didn't find really any modules 
that would suit this purpose)?

Here is what prints out:

*3-element Array{Array{Float64,1},1}:*

* [50.0]   *

* [40.0,60.0]  *

* [32.01,48.0,72.0]*

I suppose also I could default the [1][1] state to the spot price (S0), so 
then I don't need to alter the looping range and instead start the range at 
2.

Thanks!

Chris

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Andrew]

I don't think performance is the only thing Julia has going for it over 
Matlab. Julia has multiple dispatch, a sophisticated type system, macros, 
and functions can modify arrays and other mutable objects. I'm unaware of 
any plan for Matlab to add these things--they would be major changes and 
possibly very confusing for long-time users.

On Friday, October 23, 2015 at 7:18:43 AM UTC-4, Kris De Meyer wrote:
>
> On a slightly different note, in 2 or 3 release cycles, Matlab will have 
> caught up on any performance gains Julia may have introduced (by using the 
> same LLVM compiler procedures Julia uses) and then the only thing Julia 
> will have going for it is that it's free. But my cost to my employers is 
> such that if I lose as little as 3 days a year on compatibility issues, 
> they would be better off paying for a Matlab license...
>
> Best.
>
> Kris
>
>
>
>
>
>  
>
> On Thursday, October 22, 2015 at 10:58:30 PM UTC+1, Stefan Karpinski wrote:
>>
>> You can try using @code_warntype to see if there are type instabilities.
>>
>> On Thu, Oct 22, 2015 at 5:50 PM, Gunnar Farnebäck  
>> wrote:
>>
>>> If you don't have deprecation warnings I would suspect some change in 
>>> 0.4 has introduced type instabilities. If you are using typed 
>>> concatenations you could be hit by 
>>> https://github.com/JuliaLang/julia/issues/13254.
>>>
>>>
>>> Den torsdag 22 oktober 2015 kl. 23:03:00 UTC+2 skrev Kris De Meyer:

 Are there any general style guidelines for moving code from 0.3.11 to 
 0.4.0? Running the unit and functionality tests for a module that I 
 developed under 0.3.11 in 0.4, I experience a 500 times slowdown of blocks 
 of code that I time with @time. 

 Can't even imagine where I have to start looking, and find it 
 flabbergasting that perfectly valid julia code under 0.3.11 (not 
 generating 
 a single warning) can show such a performance degradation under 0.4.0.

 Anyone seen anything similar? Is there some fundamental difference in 
 how code is JIT-compiled under 0.4.0?

 Thanks,

 Kris




>>

Re: [julia-users] Got myself in another awkward git situation with Pkg.checkout

[Douglas Bates]

Thank you very much, Isaiah.  If we meet face-to-face remind me that I owe 
you a beer - probably more than one in fact.

On Friday, October 23, 2015 at 4:24:39 PM UTC-5, Isaiah wrote:
>
> (you can do `git show HASH` for each of the hashes in the log, hopefully 
> find the one that has your work, and then `git checkout HASH` to get back 
> to it. then give it a name by creating a new branch, merging it, etc.)
>
> On Fri, Oct 23, 2015 at 5:20 PM, Isaiah Norton  > wrote:
>
>> If sounds like you committed to the detached head, and if so, the commit 
>> should be a few steps back in the reflog: `git reflog`
>>
>> On Fri, Oct 23, 2015 at 5:14 PM, Douglas Bates > > wrote:
>>
>>> I tagged a new release of one of my packages.  Just to ensure that 
>>> everything was working as I thought I ran
>>>
>>> Pkg.free("MyPackage")
>>>
>>> so I would be sure to be testing on a clean copy of that release.  Of 
>>> course that meant that my local copy of the package was a detached HEAD.
>>>
>>> I then edited the code, adding a lot of new capabilities, did a local 
>>> commit of the files, tried to push these changes, found that I was on a 
>>> detached HEAD.  At that point I should have started being cautious but I 
>>> didn't.  I ran
>>>
>>> Pkg.checkout("MyPackage")
>>>
>>> and it seems that I have managed to lose all the work I did today.  Is 
>>> there some way I can get back the changes that I made on the detached HEAD?
>>>
>>
>>
>

Re: [julia-users] Re: Code runs 500 times slower under 0.4.0

[Miguel Bazdresch]

These are my thoughts exactly. I used Matlab because it was a convenient,
easy way to get results, but the warts in the design are real. Julia,
besides being convenient and easy, is a pleasure to program in.

-- mb

On Fri, Oct 23, 2015 at 5:41 PM, Andrew  wrote:

> I don't think performance is the only thing Julia has going for it over
> Matlab. Julia has multiple dispatch, a sophisticated type system, macros,
> and functions can modify arrays and other mutable objects. I'm unaware of
> any plan for Matlab to add these things--they would be major changes and
> possibly very confusing for long-time users.
>
> On Friday, October 23, 2015 at 7:18:43 AM UTC-4, Kris De Meyer wrote:
>>
>> On a slightly different note, in 2 or 3 release cycles, Matlab will have
>> caught up on any performance gains Julia may have introduced (by using the
>> same LLVM compiler procedures Julia uses) and then the only thing Julia
>> will have going for it is that it's free. But my cost to my employers is
>> such that if I lose as little as 3 days a year on compatibility issues,
>> they would be better off paying for a Matlab license...
>>
>> Best.
>>
>> Kris
>>
>>
>>
>>
>>
>>
>>
>> On Thursday, October 22, 2015 at 10:58:30 PM UTC+1, Stefan Karpinski
>> wrote:
>>>
>>> You can try using @code_warntype to see if there are type instabilities.
>>>
>>> On Thu, Oct 22, 2015 at 5:50 PM, Gunnar Farnebäck >> > wrote:
>>>
 If you don't have deprecation warnings I would suspect some change in
 0.4 has introduced type instabilities. If you are using typed
 concatenations you could be hit by
 https://github.com/JuliaLang/julia/issues/13254.


 Den torsdag 22 oktober 2015 kl. 23:03:00 UTC+2 skrev Kris De Meyer:
>
> Are there any general style guidelines for moving code from 0.3.11 to
> 0.4.0? Running the unit and functionality tests for a module that I
> developed under 0.3.11 in 0.4, I experience a 500 times slowdown of blocks
> of code that I time with @time.
>
> Can't even imagine where I have to start looking, and find it
> flabbergasting that perfectly valid julia code under 0.3.11 (not 
> generating
> a single warning) can show such a performance degradation under 0.4.0.
>
> Anyone seen anything similar? Is there some fundamental difference in
> how code is JIT-compiled under 0.4.0?
>
> Thanks,
>
> Kris
>
>
>
>
>>>

[julia-users] Building a Binomial Tree

[Christopher Alexander]

Hello all,

I am trying to write a method that builds a binomial tree for option 
pricing.  I am trying to set up my tree like this:

function build_tree(s_opt::StockOption)
  u = 1 + s_opt.pu
  d = 1 - s_opt.pd
  # qu = (exp((s_opt.r - s_opt.div) * s_opt.dt) - d) / (u - d)
  # qd = 1 - qu

  # init array
   stockTree = Vector{Float64}[ zeros(m) for m = 1:s_opt.N + 1]

  for i = 1:s_opt.N + 1
for j = 1:i
  stockTree[i][j] = s_opt.S0 * u ^ (j-1) * d ^ (i - j)
end
  end

  return stockTree
end

Is this the most "Julian" way to do this (I didn't find really any modules 
that would suit this purpose)?

Here is what prints out:

*3-element Array{Array{Float64,1},1}:*

* [50.0]   *

* [40.0,60.0]  *

* [32.01,48.0,72.0]*

I suppose also I could default the [1][1] state to the spot price (S0), so 
then I don't need to alter the looping range and instead start the range at 
2.

Thanks!

Chris

Re: [julia-users] Got myself in another awkward git situation with Pkg.checkout

[Isaiah Norton]

If sounds like you committed to the detached head, and if so, the commit
should be a few steps back in the reflog: `git reflog`

On Fri, Oct 23, 2015 at 5:14 PM, Douglas Bates  wrote:

> I tagged a new release of one of my packages.  Just to ensure that
> everything was working as I thought I ran
>
> Pkg.free("MyPackage")
>
> so I would be sure to be testing on a clean copy of that release.  Of
> course that meant that my local copy of the package was a detached HEAD.
>
> I then edited the code, adding a lot of new capabilities, did a local
> commit of the files, tried to push these changes, found that I was on a
> detached HEAD.  At that point I should have started being cautious but I
> didn't.  I ran
>
> Pkg.checkout("MyPackage")
>
> and it seems that I have managed to lose all the work I did today.  Is
> there some way I can get back the changes that I made on the detached HEAD?
>

Re: [julia-users] C-style casts

[Davide Lasagna]

Where is this documented?

[julia-users] C-style casts

[Damien]

How do I cast a UInt64 to a UInt32, discarding the extra bits, and with no 
error checking (so it can work inside @simd loops)?

There are algorithms that require this, eg. 
http://www.pcg-random.org/download.html

Re: [julia-users] Got myself in another awkward git situation with Pkg.checkout

[Isaiah Norton]

(you can do `git show HASH` for each of the hashes in the log, hopefully
find the one that has your work, and then `git checkout HASH` to get back
to it. then give it a name by creating a new branch, merging it, etc.)

On Fri, Oct 23, 2015 at 5:20 PM, Isaiah Norton 
wrote:

> If sounds like you committed to the detached head, and if so, the commit
> should be a few steps back in the reflog: `git reflog`
>
> On Fri, Oct 23, 2015 at 5:14 PM, Douglas Bates  wrote:
>
>> I tagged a new release of one of my packages.  Just to ensure that
>> everything was working as I thought I ran
>>
>> Pkg.free("MyPackage")
>>
>> so I would be sure to be testing on a clean copy of that release.  Of
>> course that meant that my local copy of the package was a detached HEAD.
>>
>> I then edited the code, adding a lot of new capabilities, did a local
>> commit of the files, tried to push these changes, found that I was on a
>> detached HEAD.  At that point I should have started being cautious but I
>> didn't.  I ran
>>
>> Pkg.checkout("MyPackage")
>>
>> and it seems that I have managed to lose all the work I did today.  Is
>> there some way I can get back the changes that I made on the detached HEAD?
>>
>
>

Re: [julia-users] C-style casts

[Davide Lasagna]

Where is this documented?

[julia-users] Hausdorff distance

[Júlio Hoffimann]

Hi,

I want to make the Hausdorff distance 
(https://en.wikipedia.org/wiki/Hausdorff_distance) available in Julia, is 
the Distances.jl package a good fit or I should create a separate package 
just for this distance between point sets?

I can think of a very simple (naive) implementation:

using Distances

A, B # matrices which columns represent the points in the pointset
D = pairwise(Euclidean(), A, B)
daB = maximum(minimum(D,2))
dbA = maximum(minimum(D,1))
result = max(daB, dbA)

-Júlio

[julia-users] Re: map() vs broadcast()

[Ján Dolinský]

## 2 argument
function map!{F}(f::F, dest::AbstractArray, A::AbstractArray, B::
AbstractArray)
for i = 1:length(A)
dest[i] = f(A[i], B[i])
end
return dest
end

The above is the map!() implementation in abstractarray.jl. Should it 
return "dest" if it is an in-place function ? Is there any fundamental 
difference between my mape4a() and map!() in abstractarray.jl ?

Thanks,
Jan

Dňa piatok, 23. októbra 2015 9:30:36 UTC+2 Ján Dolinský napísal(-a):
>
> Hi Glen,
>
> Thanks for the investigation. I am afraid the for loop in map!() is not 
> the source of the issue. Consider the folowing:
>
> _f(a,f) = (a - f) / a
>
> function mape4(A, F)
> # A - actual target values
> # F - forecasts (model estimations)
>
>   tmp = similar(A)
>   map!(_f, tmp, A, F)
>   100 * sumabs(tmp) / length(A)
>
> end
>
> function mape4a(A, F)
>
> tmp = similar(A)
> for i in eachindex(A)
> tmp[i] = _f(A[i], F[i])
> end
> 100 * sumabs(tmp) / length(A)
> end
>
> @time mape4(A,F)
>   0.452273 seconds (20.00 M allocations: 343.323 MB, 9.80% gc time)
> 832.852597807525
>
> @time mape4a(A,F)
>   0.040240 seconds (7 allocations: 38.147 MB, 1.93% gc time)
> 832.852597807525
>
> The for loop in mape4a() does not do 4 * 5 milion allocations, neither 
> should do the loop in map!(). Is this possibly a bug ?
>
> Thanks,
> Jan
>
> Dňa štvrtok, 22. októbra 2015 19:43:31 UTC+2 Glen O napísal(-a):
>>
>> I'm uncertain, but I think I may have figured out what's going on.
>>
>> The hint lies in the number of allocations - map! has 20 million 
>> allocations, while broadcast! has just 5. So I had a look at how the two 
>> functions are implemented.
>>
>> map! is implemented in perhaps the simplest way you can think of - for 
>> i=1:length(A) dest[i]=f(A[i],B[i]); end - which means that it has to store 
>> four values per iteration - i, A[i], B[i], and f(A[i],B[i]). Thus, 4 times 
>> 5 million allocations.
>>
>> broadcast! is using a cache to store values, instead, and I believe it's 
>> generating instructions using a macro instead of a regular loop, thus 
>> avoiding the assignments for i. As such, it doesn't need to store anything 
>> except for the initial caches, and after that it just overwrites the 
>> existing values. Unfortunately, that's as much as I can figure out from 
>> broadcast!, because it uses a lot of macros and a lot of relatively opaque 
>> structure.
>>
>> I'm also not entirely sure how it avoids the assignments necessary in the 
>> function call.
>>
>> On Friday, 23 October 2015 01:54:14 UTC+10, Ján Dolinský wrote:
>>>
>>> Hi,
>>>
>>> I am exploring Julia's map() and broadcast() functions. I did a simple 
>>> implementation of MAPE (mean absolute percentage error) using broadcast() 
>>> and map(). Interestingly, the difference in performance was huge.
>>>
>>> A = rand(5_000_000)
>>> F = rand(5_000_000)
>>>
>>> _f(a,f) = (a - f) / a
>>>
>>> function mape3(A, F)
>>> # A - actual target values
>>> # F - forecasts (model estimations)
>>>
>>>   tmp = similar(A)
>>>   broadcast!(_f, tmp, A, F)
>>>   100 * sumabs(tmp) / length(A)
>>>
>>> end
>>>
>>> function mape4(A, F)
>>> # A - actual target values
>>> # F - forecasts (model estimations)
>>>
>>>   tmp = similar(A)
>>>   map!(_f, tmp, A, F)
>>>   100 * sumabs(tmp) / length(A)
>>>
>>> end
>>>
>>> @time mape3(A,F) # after JIT warm-up
>>>   0.038686 seconds (8 allocations: 38.147 MB, 2.25% gc time)
>>> 876.4813057521973
>>>
>>> @time mape4(A,F) # after JIT warm-up
>>>   0.457771 seconds (20.00 M allocations: 343.323 MB, 11.29% gc time)
>>> 876.4813057521973
>>>
>>> I wonder why map() is so much slower ?
>>>
>>> Thanks,
>>> Jan
>>>
>>