[julia-users] Re: Tracking down type instability

[Ben Ward]

l_value_t* %27, %jl_value_t** %47, align 8

  %48 = getelementptr inbounds %jl_value_t, %jl_value_t* %45, i64 1, i32 0

  store %jl_value_t* %23, %jl_value_t** %43, align 8

  store %jl_value_t* %23, %jl_value_t** %48, align 8

  %49 = load i64, i64* %17, align 8

  store i64 %49, i64* %14, align 8

  ret %jl_value_t* %45


fail.split:   ; preds = %top

  call void @jl_throw(%jl_value_t* inttoptr (i64 4479322120 to 
%jl_value_t*))

  unreachable


fail12:   ; preds = 
%top.top.split_crit_edge

  call void @jl_throw(%jl_value_t* inttoptr (i64 4479322120 to 
%jl_value_t*))

  unreachable


if15.lr.ph:   ; preds = 
%top.split.top.split.split_crit_edge

  %50 = getelementptr [17 x %jl_value_t*], [17 x %jl_value_t*]* %3, i64 0, 
i64 10

  %51 = getelementptr [17 x %jl_value_t*], [17 x %jl_value_t*]* %3, i64 0, 
i64 11

  %52 = getelementptr [17 x %jl_value_t*], [17 x %jl_value_t*]* %3, i64 0, 
i64 12

  %53 = bitcast %jl_value_t* %21 to i64**

  %54 = bitcast %jl_value_t* %23 to i64**

  %55 = bitcast %jl_value_t* %27 to double**

  %56 = load i64*, i64** %53, align 8

  %57 = load i64*, i64** %54, align 8

  %58 = load double*, double** %55, align 8

  br label %if15


if15: ; preds = %if15, 
%if15.lr.ph

  %"i#287.017" = phi i64 [ 0, %if15.lr.ph ], [ %67, %if15 ]

  store %jl_value_t* %21, %jl_value_t** %50, align 8

  %59 = getelementptr i64, i64* %56, i64 %"i#287.017"

  %60 = load i64, i64* %59, align 8

  %61 = sitofp i64 %60 to double

  store %jl_value_t* %23, %jl_value_t** %51, align 8

  %62 = getelementptr i64, i64* %57, i64 %"i#287.017"

  %63 = load i64, i64* %62, align 8

  %64 = sitofp i64 %63 to double

  %65 = fdiv double %61, %64

  store %jl_value_t* %27, %jl_value_t** %52, align 8

  %66 = getelementptr double, double* %58, i64 %"i#287.017"

  store double %65, double* %66, align 8

  %67 = add nuw nsw i64 %"i#287.017", 1

  %exitcond = icmp eq i64 %67, %38

  br i1 %exitcond, label %L11.loopexit, label %if15

}

Still no vector instructions with the @simd macro :/

On Friday, September 16, 2016 at 10:00:49 PM UTC+1, Kristoffer Carlsson 
wrote:
>
> What if you add @simd after @inbounds?
>
> On Friday, September 16, 2016 at 5:48:00 PM UTC+2, Ben Ward wrote:
>>
>> The code for that function not shown is rather similar to the version 
>> that is shown:
>>
>> function 
>> distance{T<:MutationType,A<:NucleotideAlphabet}(::Type{Proportion{T}}, 
>> seqs::Vector{BioSequence{A}})
>> d, l = distance(Count{T}, seqs)
>> D = Vector{Float64}(length(d))
>> @inbounds for i in 1:length(D)
>> D[i] = d[i] / l[i]
>> end
>> return D, l
>> end
>>
>> Which has a @code_warntype which seems ok:
>>
>> *julia> **@code_warntype distance(Proportion{AnyMutation}, dnas2)*
>>
>> Variables:
>>
>>   #self#::Bio.Var.#distance
>>
>>   #unused#::Type{Bio.Var.Proportion{Bio.Var.AnyMutation}}
>>
>>   seqs@_3::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}
>>
>>   d::Array{Int64,1}
>>
>>   l::Array{Int64,1}
>>
>>   #temp#@_6::Int64
>>
>>   D::Array{Float64,1}
>>
>>   #temp#@_8::Int64
>>
>>   i::Int64
>>
>>   seqs@_10::Array{Bio.Seq.DNANucleotide,2}
>>
>>
>> Body:
>>
>>   begin 
>>
>>   $(Expr(:inbounds, false))
>>
>>   # meta: location /Users/bward/.julia/v0.5/Bio/src/var/distances.jl 
>> distance 133
>>
>>   # meta: location 
>> /Users/bward/.julia/v0.5/Bio/src/var/mutation_counting.jl count_mutations 
>> 263
>>
>>   seqs@_10::Array{Bio.Seq.DNANucleotide,2} = $(Expr(:invoke, 
>> LambdaInfo for 
>> seqmatrix(::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}, 
>> ::Symbol), :(Bio.Var.seqmatrix), :(seqs@_3), :(:seq)))
>>
>>   # meta: pop location
>>
>>   # meta: pop location
>>
>>   $(Expr(:inbounds, :pop))
>>
>>   SSAValue(0) = $(Expr(:invoke, LambdaInfo for 
>> count_mutations(::Type{Bio.Var.AnyMutation}, 
>> ::Array{Bio.Seq.DNANucleotide,2}), :(Bio.Var.count_mutations), 
>> Bio.Var.AnyMutation, :(seqs@_10)))
>>
>>   #temp#@_6::Int64 = $(QuoteNode(1))
>>
>>   SSAValue(9) = (Base.getfield)(SSAValue(0),1)::Array{Int64,1}
>>
>>   SSAValue(10) = (Base.box)(Int64,(Base.add_int)(1,1))
>>
>>   d::Array{Int64,1} = SSAValue(9)
>>
>>   #temp#@_6::Int64 = SSAValue(10)
>>
>>   SSAValue(11) = (Base.getfield)(SSAValue(0),2)::Array{Int64,1}
>>
&

[julia-users] Re: Tracking down type instability

[Ben Ward]

The code for that function not shown is rather similar to the version that 
is shown:

function 
distance{T<:MutationType,A<:NucleotideAlphabet}(::Type{Proportion{T}}, 
seqs::Vector{BioSequence{A}})
d, l = distance(Count{T}, seqs)
D = Vector{Float64}(length(d))
@inbounds for i in 1:length(D)
D[i] = d[i] / l[i]
end
return D, l
end

Which has a @code_warntype which seems ok:

*julia> **@code_warntype distance(Proportion{AnyMutation}, dnas2)*

Variables:

  #self#::Bio.Var.#distance

  #unused#::Type{Bio.Var.Proportion{Bio.Var.AnyMutation}}

  seqs@_3::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}

  d::Array{Int64,1}

  l::Array{Int64,1}

  #temp#@_6::Int64

  D::Array{Float64,1}

  #temp#@_8::Int64

  i::Int64

  seqs@_10::Array{Bio.Seq.DNANucleotide,2}


Body:

  begin 

  $(Expr(:inbounds, false))

  # meta: location /Users/bward/.julia/v0.5/Bio/src/var/distances.jl 
distance 133

  # meta: location 
/Users/bward/.julia/v0.5/Bio/src/var/mutation_counting.jl count_mutations 
263

  seqs@_10::Array{Bio.Seq.DNANucleotide,2} = $(Expr(:invoke, LambdaInfo 
for seqmatrix(::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}, 
::Symbol), :(Bio.Var.seqmatrix), :(seqs@_3), :(:seq)))

  # meta: pop location

  # meta: pop location

  $(Expr(:inbounds, :pop))

  SSAValue(0) = $(Expr(:invoke, LambdaInfo for 
count_mutations(::Type{Bio.Var.AnyMutation}, 
::Array{Bio.Seq.DNANucleotide,2}), :(Bio.Var.count_mutations), 
Bio.Var.AnyMutation, :(seqs@_10)))

  #temp#@_6::Int64 = $(QuoteNode(1))

  SSAValue(9) = (Base.getfield)(SSAValue(0),1)::Array{Int64,1}

  SSAValue(10) = (Base.box)(Int64,(Base.add_int)(1,1))

  d::Array{Int64,1} = SSAValue(9)

  #temp#@_6::Int64 = SSAValue(10)

  SSAValue(11) = (Base.getfield)(SSAValue(0),2)::Array{Int64,1}

  SSAValue(12) = (Base.box)(Int64,(Base.add_int)(2,1))

  l::Array{Int64,1} = SSAValue(11)

  #temp#@_6::Int64 = SSAValue(12) # line 256:

  SSAValue(6) = (Base.arraylen)(d::Array{Int64,1})::Int64

  D::Array{Float64,1} = 
(Core.ccall)(:jl_alloc_array_1d,(Core.apply_type)(Core.Array,Float64,1)::Type{Array{Float64,1}},(Core.svec)(Core.Any,Core.Int)::SimpleVector,Array{Float64,1},0,SSAValue(6),0)::Array{Float64,1}
 
# line 257:

  $(Expr(:inbounds, true))

  SSAValue(8) = (Base.arraylen)(D::Array{Float64,1})::Int64

  SSAValue(13) = 
(Base.select_value)((Base.sle_int)(1,SSAValue(8))::Bool,SSAValue(8),(Base.box)(Int64,(Base.sub_int)(1,1)))::Int64

  #temp#@_8::Int64 = 1

  26: 

  unless (Base.box)(Base.Bool,(Base.not_int)((#temp#@_8::Int64 === 
(Base.box)(Int64,(Base.add_int)(SSAValue(13),1)))::Bool)) goto 37

  SSAValue(14) = #temp#@_8::Int64

  SSAValue(15) = (Base.box)(Int64,(Base.add_int)(#temp#@_8::Int64,1))

  i::Int64 = SSAValue(14)

  #temp#@_8::Int64 = SSAValue(15) # line 258:

  SSAValue(5) = 
(Base.box)(Base.Float64,(Base.div_float)((Base.box)(Float64,(Base.sitofp)(Float64,(Base.arrayref)(d::Array{Int64,1},i::Int64)::Int64)),(Base.box)(Float64,(Base.sitofp)(Float64,(Base.arrayref)(l::Array{Int64,1},i::Int64)::Int64

  
(Base.arrayset)(D::Array{Float64,1},SSAValue(5),i::Int64)::Array{Float64,1}

  35: 

  goto 26

  37: 

  $(Expr(:inbounds, :pop)) # line 260:

  return 
(Core.tuple)(D::Array{Float64,1},l::Array{Int64,1})::Tuple{Array{Float64,1},Array{Int64,1}}

  end::Tuple{Array{Float64,1},Array{Int64,1}}

But then again the loop in this function is also not vectorised which I 
struggle to see why... unless division can't be.

On Friday, September 16, 2016 at 3:27:47 AM UTC+1, Ralph Smith wrote:
>
> SSAValue(15) = (Base.getfield)(SSAValue(0),1)
> *::Union{Array{Float64,1},Array{Int64,1}}*
>
>
> indicates that the first element of SSAValue(0) is ambiguous. Earlier it 
> shows that this means p from
>
> p, l = distance(Proportion{AnyMutation}, seqs)
>
> which we can't analyze from what you show here.
>
> On Thursday, September 15, 2016 at 10:08:16 AM UTC-4, Ben Ward wrote:
>>
>> Hi I have two functions and a function which calls them:
>>
>> @inline function expected_distance(::Type{JukesCantor69}, p::Float64)
>> return -0.75 * log(1 - 4 * p / 3)
>> end
>>
>> @inline function variance(::Type{JukesCantor69}, p::Float64, l::Int64)
>> return p * (1 - p) / (((1 - 4 * p / 3) ^ 2) * l)
>> end
>>
>> function distance{A<:NucleotideAlphabet}(::Type{JukesCantor69}, 
>> seqs::Vector{BioSequence{A}})
>> p, l = distance(Proportion{AnyMutation}, seqs)
>> D = Vector{Float64}(length(p))
>> V = Vector{Float64}(length(p))
>> @inbounds for i in 1:length(p)
>> D[i] = expected_distance(JukesCantor69, p[i])
>> V[i] = variance(JukesCantor69, p[i], l[i])
>> end
>> return D, V
>> end
>

Re: [julia-users] Tracking down type instability

[Ben Ward]

I see, but I thought I'd like to try and remove the instability as I wonder 
if it's one of the things preventing vectorisation of the @inbounds loop in 
the `distance` function.

On Friday, September 16, 2016 at 7:54:33 AM UTC+1, Kristoffer Carlsson 
wrote:
>
> It looks fine because for both the SSAValues with "problems" you later 
> have:
>
>  p::Array{Float64,1} = SSAValue(15)
>
>  l::Array{Int64,1} = SSAValue(17)
>
> so they actually get correctly inferred.
>
> As long as your variables list and return value is ok then things are most 
> likely ok.
>
> On Friday, September 16, 2016 at 4:37:09 AM UTC+2, Yichao Yu wrote:
>>
>> On Thu, Sep 15, 2016 at 10:08 AM, Ben Ward <axolotl...@gmail.com> wrote: 
>> > Hi I have two functions and a function which calls them: 
>> > 
>> > @inline function expected_distance(::Type{JukesCantor69}, p::Float64) 
>> > return -0.75 * log(1 - 4 * p / 3) 
>> > end 
>> > 
>> > @inline function variance(::Type{JukesCantor69}, p::Float64, l::Int64) 
>> > return p * (1 - p) / (((1 - 4 * p / 3) ^ 2) * l) 
>> > end 
>> > 
>> > function distance{A<:NucleotideAlphabet}(::Type{JukesCantor69}, 
>> > seqs::Vector{BioSequence{A}}) 
>> > p, l = distance(Proportion{AnyMutation}, seqs) 
>> > D = Vector{Float64}(length(p)) 
>> > V = Vector{Float64}(length(p)) 
>> > @inbounds for i in 1:length(p) 
>> > D[i] = expected_distance(JukesCantor69, p[i]) 
>> > V[i] = variance(JukesCantor69, p[i], l[i]) 
>> > end 
>> > return D, V 
>> > end 
>> > 
>> > But I'm seeing type uncertainty: 
>> > 
>> > @code_warntype distance(JukesCantor69, dnas) 
>> > 
>> > Variables: 
>> > 
>> >   #self#::Bio.Var.#distance 
>> > 
>> >   #unused#::Type{Bio.Var.JukesCantor69} 
>> > 
>> >   seqs::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1} 
>> > 
>> >   p::Array{Float64,1} 
>> > 
>> >   l::Array{Int64,1} 
>> > 
>> >   #temp#@_6::Int64 
>> > 
>> >   D::Array{Float64,1} 
>> > 
>> >   V::Array{Float64,1} 
>> > 
>> >   #temp#@_9::Int64 
>> > 
>> >   i::Int64 
>> > 
>> > 
>> > Body: 
>> > 
>> >   begin 
>> > 
>> >   SSAValue(0) = $(Expr(:invoke, LambdaInfo for 
>> > distance(::Type{Bio.Var.Proportion{Bio.Var.AnyMutation}}, 
>> > ::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}), 
>> > :(Bio.Var.distance), Bio.Var.Proportion{Bio.Var.AnyMutation}, :(seqs))) 
>> > 
>> >   #temp#@_6::Int64 = $(QuoteNode(1)) 
>> > 
>> >   SSAValue(15) = 
>> > (Base.getfield)(SSAValue(0),1)::Union{Array{Float64,1},Array{Int64,1}} 
>>
>> I guess we should probably print ssavalue types to make this easier to 
>> analyse 
>>
>> There's likely no type instability and the warning here is just spurious 
>>
>> > 
>> >   SSAValue(16) = (Base.box)(Int64,(Base.add_int)(1,1)) 
>> > 
>> >   p::Array{Float64,1} = SSAValue(15) 
>> > 
>> >   #temp#@_6::Int64 = SSAValue(16) 
>> > 
>> >   SSAValue(17) = 
>> > (Base.getfield)(SSAValue(0),2)::Union{Array{Float64,1},Array{Int64,1}} 
>> > 
>> >   SSAValue(18) = (Base.box)(Int64,(Base.add_int)(2,1)) 
>> > 
>> >   l::Array{Int64,1} = SSAValue(17) 
>> > 
>> >   #temp#@_6::Int64 = SSAValue(18) # line 314: 
>> > 
>> >   SSAValue(7) = (Base.arraylen)(p::Array{Float64,1})::Int64 
>> > 
>> >   D::Array{Float64,1} = 
>> > 
>> (Core.ccall)(:jl_alloc_array_1d,(Core.apply_type)(Core.Array,Float64,1)::Type{Array{Float64,1}},(Core.svec)(Core.Any,Core.Int)::SimpleVector,Array{Float64,1},0,SSAValue(7),0)::Array{Float64,1}
>>  
>>
>> > # line 315: 
>> > 
>> >   SSAValue(9) = (Base.arraylen)(p::Array{Float64,1})::Int64 
>> > 
>> >   V::Array{Float64,1} = 
>> > 
>> (Core.ccall)(:jl_alloc_array_1d,(Core.apply_type)(Core.Array,Float64,1)::Type{Array{Float64,1}},(Core.svec)(Core.Any,Core.Int)::SimpleVector,Array{Float64,1},0,SSAValue(9),0)::Array{Float64,1}
>>  
>>
>> > # line 316: 
>> > 
>> >   $(Expr(:inbounds, true)) 
>> > 
>> >   SSAValue(11) = (Base.arraylen)(p::Array{Float64,1})::Int64 
>> > 
>> >   SSAValue(19) = 
>>

[julia-users] Tracking down type instability

[Ben Ward]

Hi I have two functions and a function which calls them:

@inline function expected_distance(::Type{JukesCantor69}, p::Float64)
return -0.75 * log(1 - 4 * p / 3)
end

@inline function variance(::Type{JukesCantor69}, p::Float64, l::Int64)
return p * (1 - p) / (((1 - 4 * p / 3) ^ 2) * l)
end

function distance{A<:NucleotideAlphabet}(::Type{JukesCantor69}, 
seqs::Vector{BioSequence{A}})
p, l = distance(Proportion{AnyMutation}, seqs)
D = Vector{Float64}(length(p))
V = Vector{Float64}(length(p))
@inbounds for i in 1:length(p)
D[i] = expected_distance(JukesCantor69, p[i])
V[i] = variance(JukesCantor69, p[i], l[i])
end
return D, V
end

But I'm seeing type uncertainty:

*@code_warntype distance(JukesCantor69, dnas)*

Variables:

  #self#::Bio.Var.#distance

  #unused#::Type{Bio.Var.JukesCantor69}

  seqs::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}

  p::Array{Float64,1}

  l::Array{Int64,1}

  #temp#@_6::Int64

  D::Array{Float64,1}

  V::Array{Float64,1}

  #temp#@_9::Int64

  i::Int64


Body:

  begin 

  SSAValue(0) = $(Expr(:invoke, LambdaInfo for 
distance(::Type{Bio.Var.Proportion{Bio.Var.AnyMutation}}, 
::Array{Bio.Seq.BioSequence{Bio.Seq.DNAAlphabet{4}},1}), 
:(Bio.Var.distance), Bio.Var.Proportion{Bio.Var.AnyMutation}, :(seqs)))

  #temp#@_6::Int64 = $(QuoteNode(1))

  SSAValue(15) = (Base.getfield)(SSAValue(0),1)
*::Union{Array{Float64,1},Array{Int64,1}}*

  SSAValue(16) = (Base.box)(Int64,(Base.add_int)(1,1))

  p::Array{Float64,1} = SSAValue(15)

  #temp#@_6::Int64 = SSAValue(16)

  SSAValue(17) = (Base.getfield)(SSAValue(0),2)
*::Union{Array{Float64,1},Array{Int64,1}}*

  SSAValue(18) = (Base.box)(Int64,(Base.add_int)(2,1))

  l::Array{Int64,1} = SSAValue(17)

  #temp#@_6::Int64 = SSAValue(18) # line 314:

  SSAValue(7) = (Base.arraylen)(p::Array{Float64,1})::Int64

  D::Array{Float64,1} = 
(Core.ccall)(:jl_alloc_array_1d,(Core.apply_type)(Core.Array,Float64,1)::Type{Array{Float64,1}},(Core.svec)(Core.Any,Core.Int)::SimpleVector,Array{Float64,1},0,SSAValue(7),0)::Array{Float64,1}
 
# line 315:

  SSAValue(9) = (Base.arraylen)(p::Array{Float64,1})::Int64

  V::Array{Float64,1} = 
(Core.ccall)(:jl_alloc_array_1d,(Core.apply_type)(Core.Array,Float64,1)::Type{Array{Float64,1}},(Core.svec)(Core.Any,Core.Int)::SimpleVector,Array{Float64,1},0,SSAValue(9),0)::Array{Float64,1}
 
# line 316:

  $(Expr(:inbounds, true))

  SSAValue(11) = (Base.arraylen)(p::Array{Float64,1})::Int64

  SSAValue(19) = 
(Base.select_value)((Base.sle_int)(1,SSAValue(11))::Bool,SSAValue(11),(Base.box)(Int64,(Base.sub_int)(1,1)))::Int64

  #temp#@_9::Int64 = 1

  22: 

  unless (Base.box)(Base.Bool,(Base.not_int)((#temp#@_9::Int64 === 
(Base.box)(Int64,(Base.add_int)(SSAValue(19),1)))::Bool)) goto 43

  SSAValue(20) = #temp#@_9::Int64

  SSAValue(21) = (Base.box)(Int64,(Base.add_int)(#temp#@_9::Int64,1))

  i::Int64 = SSAValue(20)

  #temp#@_9::Int64 = SSAValue(21) # line 317:

  SSAValue(12) = (Base.arrayref)(p::Array{Float64,1},i::Int64)::Float64

  $(Expr(:inbounds, false))

  # meta: location /Users/bward/.julia/v0.5/Bio/src/var/distances.jl 
expected_distance 69

  SSAValue(13) = $(Expr(:invoke, LambdaInfo for log(::Float64), 
:(Bio.Var.log), 
:((Base.box)(Base.Float64,(Base.sub_float)((Base.box)(Float64,(Base.sitofp)(Float64,1)),(Base.box)(Base.Float64,(Base.div_float)((Base.box)(Base.Float64,(Base.mul_float)((Base.box)(Float64,(Base.sitofp)(Float64,4)),SSAValue(12))),(Base.box)(Float64,(Base.sitofp)(Float64,3)

  # meta: pop location

  $(Expr(:inbounds, :pop))

  SSAValue(5) = 
(Base.box)(Base.Float64,(Base.mul_float)(-0.75,SSAValue(13)))

  
(Base.arrayset)(D::Array{Float64,1},SSAValue(5),i::Int64)::Array{Float64,1} 
# line 318:

  SSAValue(14) = (Base.arrayref)(p::Array{Float64,1},i::Int64)::Float64

  SSAValue(6) = 
(Base.box)(Base.Float64,(Base.div_float)((Base.box)(Base.Float64,(Base.mul_float)(SSAValue(14),(Base.box)(Base.Float64,(Base.sub_float)((Base.box)(Float64,(Base.sitofp)(Float64,1)),SSAValue(14),(Base.box)(Base.Float64,(Base.mul_float)((Base.Math.box)(Base.Math.Float64,(Base.Math.powi_llvm)((Base.box)(Base.Float64,(Base.sub_float)((Base.box)(Float64,(Base.sitofp)(Float64,1)),(Base.box)(Base.Float64,(Base.div_float)((Base.box)(Base.Float64,(Base.mul_float)((Base.box)(Float64,(Base.sitofp)(Float64,4)),SSAValue(14))),(Base.box)(Float64,(Base.sitofp)(Float64,3)),(Base.box)(Int32,(Base.checked_trunc_sint)(Int32,2::Float64,(Base.box)(Float64,(Base.sitofp)(Float64,(Base.arrayref)(l::Array{Int64,1},i::Int64)::Int64))

  
(Base.arrayset)(V::Array{Float64,1},SSAValue(6),i::Int64)::Array{Float64,1}

  41: 

  goto 22

  43: 

  $(Expr(:inbounds, :pop)) # line 320:

  return 
(Core.tuple)(D::Array{Float64,1},V::Array{Float64,1})::Tuple{Array{Float64,1},Array{Float64,1}}

  

[julia-users] Re: Telling if code is vectorised.

[Ben Ward]

Ahah! I completely forgot about warntype! Thanks!

Out of interest, as I've only started to read llvm and native instructions 
generated by my julia functions, is there a way to tell from the llvm if 
there's type instability - say one forgets to do a code_warntype, are there 
llvm instructions that when you see them, you should go "Doh man I need to 
do @code_warntype as there's something not right here"?

On Wednesday, September 14, 2016 at 4:04:29 PM UTC+1, Kristoffer Carlsson 
wrote:
>
> Your code is type unstable. Use a = UInt64(0).
>
> On Wednesday, September 14, 2016 at 4:33:39 PM UTC+2, Ben Ward wrote:
>>
>> Hi,
>>
>> I've written a simple function:
>>
>> function testfun2()
>> a = 0
>> @inbounds @simd for i in UInt64(1):UInt64(1000)
>> i = i - ((i >> 1) & 0x)
>> a += ((i & 0x) + ((i >> 2) & 0x))
>> end
>> return a
>> end
>>
>> I applies the same set of bit operations to a series of UInt64's and 
>> accumulates a result.
>>
>> And know from the Intel blog on vectorisation, in the llvm code generated 
>> by julia there is a set of instructions to look out for that indicate 
>> vectorised code: vector.head and vector.ph:
>>
>> I'm wondering why I don't see those instructions in the llvm generated 
>> for this function, I don't think I've violated any of the rules of writing 
>> loops that can be vectorised:
>>
>> *julia> **@code_llvm testfun2()*
>>
>>
>> define %jl_value_t* @julia_testfun2_70900() #0 {
>>
>> top:
>>
>>   %0 = call %jl_value_t*** @jl_get_ptls_states() #1
>>
>>   %1 = alloca [11 x %jl_value_t*], align 8
>>
>>   %.sub = getelementptr inbounds [11 x %jl_value_t*], [11 x 
>> %jl_value_t*]* %1, i64 0, i64 0
>>
>>   %2 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
>> i64 8
>>
>>   %3 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
>> i64 2
>>
>>   %a = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
>> i64 7
>>
>>   %4 = bitcast %jl_value_t** %2 to i8*
>>
>>   call void @llvm.memset.p0i8.i32(i8* %4, i8 0, i32 24, i32 8, i1 false)
>>
>>   %5 = bitcast [11 x %jl_value_t*]* %1 to i64*
>>
>>   %6 = bitcast %jl_value_t** %3 to i8*
>>
>>   call void @llvm.memset.p0i8.i64(i8* %6, i8 0, i64 40, i32 8, i1 false)
>>
>>   store i64 18, i64* %5, align 8
>>
>>   %7 = bitcast %jl_value_t*** %0 to i64*
>>
>>   %8 = load i64, i64* %7, align 8
>>
>>   %9 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
>> i64 1
>>
>>   %10 = bitcast %jl_value_t** %9 to i64*
>>
>>   store i64 %8, i64* %10, align 8
>>
>>   store %jl_value_t** %.sub, %jl_value_t*** %0, align 8
>>
>>   %"r#274" = alloca %UnitRange.6, align 8
>>
>>   store %jl_value_t* inttoptr (i64 4400218208 to %jl_value_t*), 
>> %jl_value_t** %a, align 8
>>
>>   %11 = getelementptr inbounds %UnitRange.6, %UnitRange.6* %"r#274", i64 
>> 0, i32 0
>>
>>   store i64 1, i64* %11, align 8
>>
>>   %12 = getelementptr inbounds %UnitRange.6, %UnitRange.6* %"r#274", i64 
>> 0, i32 1
>>
>>   store i64 1000, i64* %12, align 8
>>
>>   %13 = call i64 @julia_simd_inner_length_70896(%UnitRange.6* nonnull 
>> %"r#274", i64 0) #0
>>
>>   %14 = icmp eq i64 %13, 0
>>
>>   br i1 %14, label %L.backedge, label %if26.lr.ph
>>
>>
>> L8:   ; preds = %if26
>>
>>   %15 = load %jl_value_t*, %jl_value_t** %a, align 8
>>
>>   store %jl_value_t* %15, %jl_value_t** %40, align 8
>>
>>   %16 = getelementptr inbounds %jl_value_t, %jl_value_t* %15, i64 -1, i32 
>> 0
>>
>>   %17 = bitcast %jl_value_t** %16 to i64*
>>
>>   %18 = load i64, i64* %17, align 8
>>
>>   %19 = and i64 %18, -16
>>
>>   %20 = inttoptr i64 %19 to %jl_value_t*
>>
>>   %21 = icmp eq %jl_value_t* %20, inttoptr (i64 4400088496 to 
>> %jl_value_t*)
>>
>>   br i1 %21, label %L11, label %L10
>>
>>
>> L10:  ; preds = %L8
>>
>>   %22 = load i64, i64* %46, align 8
>>
>>   store i64 %22, i64* %47, align 8
>>
>>   %23 = and i64 %53, 3689348814741910323
>>
>>   %24 = lshr i64 %53, 2
>>
>>   %25 = and i64 %24, 3689348814741910323
&g

[julia-users] Telling if code is vectorised.

[Ben Ward]

Hi,

I've written a simple function:

function testfun2()
a = 0
@inbounds @simd for i in UInt64(1):UInt64(1000)
i = i - ((i >> 1) & 0x)
a += ((i & 0x) + ((i >> 2) & 0x))
end
return a
end

I applies the same set of bit operations to a series of UInt64's and 
accumulates a result.

And know from the Intel blog on vectorisation, in the llvm code generated 
by julia there is a set of instructions to look out for that indicate 
vectorised code: vector.head and vector.ph:

I'm wondering why I don't see those instructions in the llvm generated for 
this function, I don't think I've violated any of the rules of writing 
loops that can be vectorised:

*julia> **@code_llvm testfun2()*


define %jl_value_t* @julia_testfun2_70900() #0 {

top:

  %0 = call %jl_value_t*** @jl_get_ptls_states() #1

  %1 = alloca [11 x %jl_value_t*], align 8

  %.sub = getelementptr inbounds [11 x %jl_value_t*], [11 x %jl_value_t*]* 
%1, i64 0, i64 0

  %2 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
i64 8

  %3 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
i64 2

  %a = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
i64 7

  %4 = bitcast %jl_value_t** %2 to i8*

  call void @llvm.memset.p0i8.i32(i8* %4, i8 0, i32 24, i32 8, i1 false)

  %5 = bitcast [11 x %jl_value_t*]* %1 to i64*

  %6 = bitcast %jl_value_t** %3 to i8*

  call void @llvm.memset.p0i8.i64(i8* %6, i8 0, i64 40, i32 8, i1 false)

  store i64 18, i64* %5, align 8

  %7 = bitcast %jl_value_t*** %0 to i64*

  %8 = load i64, i64* %7, align 8

  %9 = getelementptr [11 x %jl_value_t*], [11 x %jl_value_t*]* %1, i64 0, 
i64 1

  %10 = bitcast %jl_value_t** %9 to i64*

  store i64 %8, i64* %10, align 8

  store %jl_value_t** %.sub, %jl_value_t*** %0, align 8

  %"r#274" = alloca %UnitRange.6, align 8

  store %jl_value_t* inttoptr (i64 4400218208 to %jl_value_t*), 
%jl_value_t** %a, align 8

  %11 = getelementptr inbounds %UnitRange.6, %UnitRange.6* %"r#274", i64 0, 
i32 0

  store i64 1, i64* %11, align 8

  %12 = getelementptr inbounds %UnitRange.6, %UnitRange.6* %"r#274", i64 0, 
i32 1

  store i64 1000, i64* %12, align 8

  %13 = call i64 @julia_simd_inner_length_70896(%UnitRange.6* nonnull 
%"r#274", i64 0) #0

  %14 = icmp eq i64 %13, 0

  br i1 %14, label %L.backedge, label %if26.lr.ph


L8:   ; preds = %if26

  %15 = load %jl_value_t*, %jl_value_t** %a, align 8

  store %jl_value_t* %15, %jl_value_t** %40, align 8

  %16 = getelementptr inbounds %jl_value_t, %jl_value_t* %15, i64 -1, i32 0

  %17 = bitcast %jl_value_t** %16 to i64*

  %18 = load i64, i64* %17, align 8

  %19 = and i64 %18, -16

  %20 = inttoptr i64 %19 to %jl_value_t*

  %21 = icmp eq %jl_value_t* %20, inttoptr (i64 4400088496 to %jl_value_t*)

  br i1 %21, label %L11, label %L10


L10:  ; preds = %L8

  %22 = load i64, i64* %46, align 8

  store i64 %22, i64* %47, align 8

  %23 = and i64 %53, 3689348814741910323

  %24 = lshr i64 %53, 2

  %25 = and i64 %24, 3689348814741910323

  %26 = add nuw nsw i64 %25, %23

  store %jl_value_t* inttoptr (i64 4408276712 to %jl_value_t*), 
%jl_value_t** %2, align 8

  %27 = call %jl_value_t* @jl_box_uint64(i64 zeroext %26)

  store %jl_value_t* %27, %jl_value_t** %44, align 8

  %28 = call %jl_value_t* @jl_apply_generic(%jl_value_t** %2, i32 3)

  store %jl_value_t* %28, %jl_value_t** %41, align 8

  br label %L12


L11:  ; preds = %L8, %if26

  %"#temp#3.0" = phi %jl_value_t* [ inttoptr (i64 4409008592 to 
%jl_value_t*), %if26 ], [ inttoptr (i64 4423075536 to %jl_value_t*), %L8 ]

  store %jl_value_t* %"#temp#3.0", %jl_value_t** %42, align 8

  %29 = load i64, i64* %46, align 8

  store i64 %29, i64* %47, align 8

  %30 = and i64 %53, 3689348814741910323

  %31 = lshr i64 %53, 2

  %32 = and i64 %31, 3689348814741910323

  %33 = add nuw nsw i64 %32, %30

  store %jl_value_t* inttoptr (i64 4408276712 to %jl_value_t*), 
%jl_value_t** %2, align 8

  %34 = call %jl_value_t* @jl_box_uint64(i64 zeroext %33)

  store %jl_value_t* %34, %jl_value_t** %44, align 8

  %35 = call %jl_value_t* @jl_invoke(%jl_value_t* %"#temp#3.0", 
%jl_value_t** %2, i32 3)

  store %jl_value_t* %35, %jl_value_t** %43, align 8

  br label %L12


L12:  ; preds = %L11, %L10

  %storemerge.in.in = phi %jl_value_t* [ %28, %L10 ], [ %35, %L11 ]

  %storemerge.in = bitcast %jl_value_t* %storemerge.in.in to i64*

  %storemerge = load i64, i64* %storemerge.in, align 1

  %36 = call %jl_value_t* @jl_box_uint64(i64 zeroext %storemerge)

  store %jl_value_t* %36, %jl_value_t** %a, align 8

  %37 = add nuw i64 %"i#277.040", 1

  %exitcond = icmp eq i64 %37, %13

  br i1 %exitcond, label %L.backedge.loopexit, label %if26


L.backedge.loopexit:  ; preds = %L12

  

[julia-users] @inbounds macro scope, and @simd

[Ben Ward]

Hi, Just wondering, I have the following doube loop:

for site in 1:nsites
votes[1] = votes[2] = votes[3] = votes[4] = votes[5] = 0
for seq in 1:neqs
nuc = mat[seq, site]
votes[1] += iscompatible(nuc, DNA_A)
votes[2] += iscompatible(nuc, DNA_C)
votes[3] += iscompatible(nuc, DNA_G)
votes[4] += iscompatible(nuc, DNA_T)
votes[5] += iscompatible(nuc, DNA_Gap)
end
end

Say I add an @inbounds macro to the outer loop to eliminate bounds checks, 
will it's effects extend to setmatrix statements in the inner loop. 
Inspecting the expanded macro I believe it is the case, as an @inbounds is 
set to true, and then after the inner loop is popped. But I'm not 100% sure 
if I am correct that is indeed how it works:

*quote  # REPL[63], line 2:*

*begin *

*$(Expr(:inbounds, true))*

*for site = 1:nsites # REPL[63], line 3:*

*votes[1] = (votes[2] = (votes[3] = (votes[4] = (votes[5] = 
0 # REPL[63], line 4:*

*for seq = 1:neqs # REPL[63], line 5:*

*nuc = mat[seq,site] # REPL[63], line 6:*

*votes[1] += iscompatible(nuc,DNA_A) # REPL[63], line 7:*

*votes[2] += iscompatible(nuc,DNA_C) # REPL[63], line 8:*

*votes[3] += iscompatible(nuc,DNA_G) # REPL[63], line 9:*

*votes[4] += iscompatible(nuc,DNA_T) # REPL[63], line 10:*

*votes[5] += iscompatible(nuc,DNA_Gap)*

*end*

*end*

*$(Expr(:inbounds, :pop))*

*end*

*end*

I'd also like someone's opinion. Will I benefit from @simd on the inner 
loop?

The function `iscompatible` is annotated with @inline, and has no branching:

@inline function iscompatible{T<:Nucleotide}(x::T, y::T)
return compatbits(x) & compatbits(y) != 0
end

# Return the compatibility bits of `nt`.
@inline function compatbits(nt::Nucleotide)
return reinterpret(UInt8, nt)
end

As per the assumptions of an simd loop I read in the docs, each iteration 
is independent, order does not matter.
I'd just like some advice as if I'm right this will be the first time I use 
a simd loop to speed up my loops. 

Thanks,
Ben.

Re: [julia-users] Recursive/Circular parametric types

[Ben Ward]

Hi Tim,

That's a shame, I was hoping that doing the above would let me create 
several different concrete PhylogenyNode and PhylogenyEdge types, that can 
be used together. I guess since this is an abuse I have to pair one 
concrete PhylogenyEdge type with one concrete PhylogenyNode type? I 
wondered is a valid alternative to do something like this:

abstract AbstractNode
abstract PhylogenyNode <: AbstractNode
abstract NetworkNode <: AbstractNode
abstract AbstractEdge
abstract PhylogenyEdge <: AbstractEdge
abstract NetworkEdge <: AbstractEdge

type Branch <: PhylogenyEdge
   from::PhylogenyNode
   to::PhylogenyNode
   length::Float64

function Branch()
   x = new()
   length!(x, -1.0)
   return x
   end
end

type Clade <: PhylogenyNode
   from::PhylogenyEdge
   to::Vector{PhylogenyEdge}
   confidence::Float64

function Clade()
   x = new()
   x.to = Vector{PhylogenyEdge}()
   confidence!(x, -1.0)
   return x
   end
end

And define getters and setters in such a way that type assertions make 
things certain for the compiler?
I saw that Jeff proposed a similar solution in julia issue #269 to handle 
circular type declarations.




On Wednesday, August 24, 2016 at 4:11:06 PM UTC+1, Tim Holy wrote:
>
> I don't think that's type-stable. Since each node of each tree will also 
> be a 
> different type, I also think you'll end up hating life due to compile 
> times. 
> There's some (peripherally) relevant discussion at http://docs.julialang.org/ 
>
> en/latest/manual/performance-tips/#the-dangers-of-abusing-multiple-dispatch- 
>
> aka-more-on-types-with-values-as-parameters 
> <http://docs.julialang.org/en/latest/manual/performance-tips/#the-dangers-of-abusing-multiple-dispatch-aka-more-on-types-with-values-as-parameters>
>  
>
> Best, 
> --Tim 
>
> On Tuesday, August 23, 2016 3:28:17 PM CDT Ben Ward wrote: 
> > I'm doing some development and wondered if this kind of pattern is 
> > problematic: 
> > 
> > abstract AbstractNode 
> > abstract PhylogenyNode <: AbstractNode 
> > abstract NetworkNode <: AbstractNode 
> > abstract AbstractEdge 
> > abstract PhylogenyEdge <: AbstractEdge 
> > abstract NetworkEdge <: AbstractEdge 
> > 
> > type Branch{N <: PhylogenyNode} <: PhylogenyEdge 
> > from::N{Branch} 
> > to::N{Branch} 
> > length::Float64 
> > 
> > function Branch{N}(::Type{N}) 
> > x = new() 
> > length!(x, -1.0) 
> > return x 
> > end 
> > end 
> > 
> > type Clade{E <: PhylogenyEdge} <: PhylogenyNode 
> > from::E 
> > to::Vector{E} 
> > confidence::Float64 
> > 
> > function Clade{E}(::Type{E}) 
> > x = new() 
> > x.to = Vector{E}() 
> > confidence!(x, -1.0) 
> > return x 
> > end 
> > end 
> > 
> > 
> > 
> > As you can see both concrete types are parametric, and as a result there 
> is 
> > a certain circularity to it 
> > Clade{Branch{Clade{Branch{Clade{Branch} That ultimately ends in 
> > something like Clade{Branch{N<:PhylogenyNode}}. I'd like to know if this 
> is 
> > type-certain or not - the fact it terminates in N<:PhylogenyNode or 
> > E<:PhylogenyEdge makes me doubt it. 
>
>
>

[julia-users] Recursive/Circular parametric types

[Ben Ward]

I'm doing some development and wondered if this kind of pattern is 
problematic:

abstract AbstractNode
abstract PhylogenyNode <: AbstractNode
abstract NetworkNode <: AbstractNode
abstract AbstractEdge
abstract PhylogenyEdge <: AbstractEdge
abstract NetworkEdge <: AbstractEdge

type Branch{N <: PhylogenyNode} <: PhylogenyEdge
from::N{Branch}
to::N{Branch}
length::Float64

function Branch{N}(::Type{N})
x = new()
length!(x, -1.0)
return x
end
end

type Clade{E <: PhylogenyEdge} <: PhylogenyNode
from::E
to::Vector{E}
confidence::Float64

function Clade{E}(::Type{E})
x = new()
x.to = Vector{E}()
confidence!(x, -1.0)
return x
end
end



As you can see both concrete types are parametric, and as a result there is 
a certain circularity to it 
Clade{Branch{Clade{Branch{Clade{Branch} That ultimately ends in 
something like Clade{Branch{N<:PhylogenyNode}}. I'd like to know if this is 
type-certain or not - the fact it terminates in N<:PhylogenyNode or 
E<:PhylogenyEdge makes me doubt it.

[julia-users] Perhaps stupid question: Method type stability and throws

[Ben Ward]

Hi,

I have a question which may be laughable to CS people (be gentle, I'm a 
Biologist), but we know from guidelines that Julia performs best, when a 
method always returns the same type of value.
So the Julia type inference knows - ok, use this method with an Int, it 
will always return a bool say. How does throws fit into this? To my mind 
this means the method used with an Int may return a bool, but it also may 
result in a type of (e.g.) ArgumentError getting thrown. So how do throws 
in functions affect performance and julian's ability to infer types and 
optimise code (if at all?)? 

Thanks,
Ben W.

[julia-users] Re: Trouble with parametric parts of a type def

[Ben Ward]

The only way I've come up with so far is with this clunky looking type 
union:

intsAndVecs = Union{Unsigned,
AbstractVector{UInt8},
AbstractVector{UInt16},
AbstractVector{UInt32},
AbstractVector{UInt64},
AbstractVector{UInt128}}


On Thursday, April 28, 2016 at 11:44:14 PM UTC+1, Ben Ward wrote:
>
> Hi, I'm tying to create a parametric type which can hold a variable which 
> could be any kind of vector, and whatever the vector is, it can store one 
> of the UInts. So it could have:
>
> UInt8[1,2,3]
> UInt64[1,2,3]
> UnitRange(1, 3)
> UnitRange(UInt16(1), UInt16(3))
> ... and so on.
>
> But I'm having trouble deciding what needs to go in the parametric part of 
> it's definition: e.g.
>
> MyType{T <: (unsure of this bit)}
> val::T
> end
>
>
> I thought that AbstractVector{Unsigned} might be the way to go, but 
> Vector{UInt8} does not inherit from AbstractVector{Unsigned}.
>
> Thanks,
> Ben.
>

[julia-users] Trouble with parametric parts of a type def

[Ben Ward]

Hi, I'm tying to create a parametric type which can hold a variable which 
could be any kind of vector, and whatever the vector is, it can store one 
of the UInts. So it could have:

UInt8[1,2,3]
UInt64[1,2,3]
UnitRange(1, 3)
UnitRange(UInt16(1), UInt16(3))
... and so on.

But I'm having trouble deciding what needs to go in the parametric part of 
it's definition: e.g.

MyType{T <: (unsure of this bit)}
val::T
end


I thought that AbstractVector{Unsigned} might be the way to go, but 
Vector{UInt8} does not inherit from AbstractVector{Unsigned}.

Thanks,
Ben.

[julia-users] Julia programs in pipes

[Ben Ward]

Hi,

I'm wondering how easy it is to make a Julia script that can be part of a 
shell pipeline `cat file > julia myscript.jl > output.txt`?

When reading the documentation, I get the impression most io is designed to 
be done all at once. I know how to loop over a file handle or stream, and I 
think
STDOUT and STDIN are accessible from Julia, I'm just unsure about how to 
make it act as one link in a longer shell pipe.
For example, if I were to loop using a stream of STDIN, waiting for a piece 
of data from the output of previous step instead of just thinking STDIN is 
finished or just waiting for all of it,
I'd like things to move through piecemeal and so on, like a node stream or 
unix pipe does. Are there any good examples of people doing this?

Until now I've just read in whole files or iterated over lines of files, 
and analysed them.

Thanks,
Ben.

[julia-users] Re: Issue with macros for creating types

[Ben Ward]

Ohhh I see it, yep. Can't give a number in a parametric type definition - 
ugh, well that's tiredness!

On Saturday, February 27, 2016 at 2:12:43 AM UTC, Ben Ward wrote:
>
> Hi,
>
> I'm trying to create a macro for the BioJulia project which will allow 
> easy creation of biological alphabets, which will work with the new 
> BioSequence type we are designing as a major improvement to the Seq module. 
> As it will be creating types and functions, I expect hygiene will be an 
> issue, but before I even get to that, let's start simple. Just a quote, and 
> some interpolation:
>
> *macro create_alphabet(alph_name, n_bits)*
>
>*quote*
>
>*immutable $(alph_name){$(n_bits)} <: Alphabet end*
>
>*end*
>
>*end*
>
> *@create_alphabet (macro with 1 method)*
>
> *julia> **abstract Alphabet*
>
> *julia> **@create_alphabet hi 1*
>
> *ERROR: syntax: malformed expression*
>
> * in eval(::Module, ::Any) at ./boot.jl:267*
>
>
>
> For the life of me, I really cannot see what I've done wrong - it's one 
> line with two interpolated words! What massively obvious thing am I being a 
> moron about at 2 am?
>
> Thanks,
> Ben.
>

[julia-users] Issue with macros for creating types

[Ben Ward]

Hi,

I'm trying to create a macro for the BioJulia project which will allow easy 
creation of biological alphabets, which will work with the new BioSequence 
type we are designing as a major improvement to the Seq module. As it will 
be creating types and functions, I expect hygiene will be an issue, but 
before I even get to that, let's start simple. Just a quote, and some 
interpolation:

*macro create_alphabet(alph_name, n_bits)*

   *quote*

   *immutable $(alph_name){$(n_bits)} <: Alphabet end*

   *end*

   *end*

*@create_alphabet (macro with 1 method)*

*julia> **abstract Alphabet*

*julia> **@create_alphabet hi 1*

*ERROR: syntax: malformed expression*

* in eval(::Module, ::Any) at ./boot.jl:267*



For the life of me, I really cannot see what I've done wrong - it's one 
line with two interpolated words! What massively obvious thing am I being a 
moron about at 2 am?

Thanks,
Ben.

[julia-users] Julia compilation - forcing architecture

[Ben Ward]

Hi,

I'm compiling Julia on a multi node cluster system and whilst I can get to 
to compile and install successfully,
it compiles for the incorrect cpu architecture perhaps because I'm building 
it from a specific software install node.
Which Makefiles, and their variables, must I edit in order to ensure that 
it compiles for x86_64 architecture?
ARCH? MARCH? The README of the repo is slightly unclear on this to me.

Thanks,
Ben.

[julia-users] Compilation of Julia on cluster - getting some error

[Ben Ward]

Hi Julia community!

I've got Julia compiled successfully on my workstation without issue, but 
I'm trying to do it now in my cluster environment at work, which I believe 
is RedHat or some similar linux.

I have permissions to install and compile software in a given directory. 
However during compilation I get some errors to do with OpenBLAS I've not 
seen before when compiling on my workstation:

../kernel/x86_64/dgemm_kernel_4x8_haswell.S:2651: Error: no such instruction
: `vpermpd $ 0x1b,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x8_haswell.S:2651: Error: no such 
instruction: `vpermpd $ 0x1b,%ymm3,%ymm3'
../kernel/x86_64/dgemm_kernel_4x8_haswell.S:2651: Error: no such 
instruction: `vpermpd $ 0xb1,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x8_haswell.S:2651: Error: no such instruction
: `vpermpd $ 0xb1,%ymm3,%ymm3'
make[3]: *** [dgemm_kernel_HASWELL.o] Error 1
make[3]: *** Waiting for unfinished jobs
make[2]: *** [libs] Error 1
\033[33;1m*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. 
Rebuild with 'make OPENBLAS_USE_THREAD=0' if OpenBLAS had trouble linking 
libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were 
errors building SandyBridge support. Both these options can also be used 
simultaneously. ***\033[0m
make[1]: *** [build/openblas/libopenblas64_.so] Error 1
make: *** [julia-deps] Error 2


I've tried make -C deps clean-openblas but I still get the error appear, 
and I don't know if what I'm seeing is OpenBLAS having trouble linking with 
libpthread.so, or if it's an error building sandybridge support - or if 
it's an issue that's neither of those two and something else entirely.
Any advice is greatly appreciated.

Many Thanks,
Ben.

Re: [julia-users] AVL Trees - SortedSet?

[Ben Ward]

Hi Kevin,

Thanks! That's one less thing for me to implement - no point in re-doing 
stuff already in .jl packages :)

On Thursday, February 4, 2016 at 12:48:50 AM UTC, Kevin Squire wrote:
>
> Not in a place where I can look at that link, but yes, SortedSet and 
> SortedDict are base on AVL trees. 
>
> Cheers,
>Kevin
>
> On Wednesday, February 3, 2016, Ben Ward <axolotl...@gmail.com 
> > wrote:
>
>> Hi,
>>
>> I'm looking to implement a C library in Julia which makes use of another 
>> library which contains a sorted dictionary called a AVL table. I don't know 
>> much about the technical internals of ALV table, and the C looks 
>> complicated to get into, if Julia already has something that does the job. 
>> Does the data structure SortedSet in DataStructures.jl perform the same 
>> functionality in a similar way? The ALV table C implementation is here: 
>> https://github.com/benedictpaten/sonLib/blob/master/C/inc/avl.h
>>
>> Thanks,
>> Ben.
>>
>

[julia-users] AVL Trees - SortedSet?

[Ben Ward]

Hi,

I'm looking to implement a C library in Julia which makes use of another 
library which contains a sorted dictionary called a AVL table. I don't know 
much about the technical internals of ALV table, and the C looks 
complicated to get into, if Julia already has something that does the job. 
Does the data structure SortedSet in DataStructures.jl perform the same 
functionality in a similar way? The ALV table C implementation is 
here: https://github.com/benedictpaten/sonLib/blob/master/C/inc/avl.h

Thanks,
Ben.

[julia-users] Anonymous functions now faster? Need for functors?

[Ben Ward]

Hi,

I just saw this merged PR .

In the past I used the functor trick of defining a type, and then a call 
method for the type, and passing this to a function, to get past the 
anonymous function inefficiency.

Does this PR mean (to a mortal like me) that on 0.5 now this is no longer 
necessary?

Thanks,
Ben.

[julia-users] A Julia equivalent of destructors - is that a thing?

[Ben Ward]

Hi,

I'm not sure of the answer, but is there an equivalent of the concept of a 
class destructor in Julia? I can imagine a situation where there is some 
type where a field is something like - for example - a file stream, and you 
want to make sure if that variable gets deleted by the gc, then the stream 
that is one of those variables get's closed.

Best,
Ben.

Re: [julia-users] A Julia equivalent of destructors - is that a thing?

[Ben Ward]

Thanks Stefan,

After googling the method I noticed some github issues where Jeff said that 
finalizer is inefficient and that we should not use it so much #11207. 
Should I be concerned about this?

The reason I ask is I'm about to try and forge ahead with BioJulia's 
Dat.jl, and I guess stream work, or types associated with a stream field 
are a possibility.

On Wednesday, December 9, 2015 at 7:16:25 PM UTC, Stefan Karpinski wrote:
>
> http://docs.julialang.org/en/release-0.4/stdlib/base/#Base.finalizer
>
> On Wed, Dec 9, 2015 at 2:12 PM, Ben Ward <axolotl...@gmail.com 
> > wrote:
>
>> Hi,
>>
>> I'm not sure of the answer, but is there an equivalent of the concept of 
>> a class destructor in Julia? I can imagine a situation where there is some 
>> type where a field is something like - for example - a file stream, and you 
>> want to make sure if that variable gets deleted by the gc, then the stream 
>> that is one of those variables get's closed.
>>
>> Best,
>> Ben.
>>
>>
>

Re: [julia-users] Get the OS

[Ben Ward]

I don't think Base.DevNull works for this at present, it only works for 
redirecting streams.

On Tuesday, December 8, 2015 at 8:54:47 PM UTC, Yichao Yu wrote:
>
> Also 
> http://julia.readthedocs.org/en/latest/stdlib/base/?highlight=devnull#Base.DevNull
>  
>
> On Tue, Dec 8, 2015 at 3:53 PM, Jacob Quinn <quinn@gmail.com 
> > wrote: 
> > See here: 
> > 
> http://docs.julialang.org/en/latest/stdlib/base/?highlight=windows#Base.@windows
>  
> > 
> > There are also the 
> > 
> > @windows_only ...code... 
> > @unix_only ...code 
> > @osx_only ...code 
> > @linux_only code... 
> > 
> > Or the variable name `OS_NAME` that will return the OS name as a symbol. 
> > 
> > Plenty of options :) 
> > 
> > On Tue, Dec 8, 2015 at 1:46 PM, Ben Ward <axolotl...@gmail.com 
> > wrote: 
> >> 
> >> Hi, 
> >> 
> >> How can I check the OS in Julia? I thought it may be in the Getting 
> Around 
> >> section of the reference manual but it's not. Basically I'd like to 
> check 
> >> the OS to see if I should write to /dev/null or just NUL in the case of 
> >> windows. 
> >> 
> >> Thanks, 
> >> Ben. 
> > 
> > 
>

Re: [julia-users] Re: The growth of Julia userbase

[Ben Ward]

Thanks, for the responses everyone, it was incredibly helpful! :)

On Thursday, November 12, 2015 at 9:21:33 AM UTC, Viral Shah wrote:
>
> Attching the github traffic data. Doesn't have history though, but the 
> numbers were bigger than what I expected when I first saw this.
>
>
> Some other interesting numbers:
>
> 1. ~10,000 pageviews on the website daily
>
> 2. 1.7M "users" and 8.6M pageviews over the lifetime of julialang.org - 
> as per google analytics
>
> 3. The mailing lists have ~4000 subscribers combined
>
> 4. Github page gets 74000 views every fortnight, ~10,000 uniques. ~3000 
> clones every fortnight, with ~700 uniques.
>
>
> Iain posted the packages data, and the github stars and watch data is 
> visible to everyone. 
>
>
> -viral
>
>
>
>
>
> <https://lh3.googleusercontent.com/-5gwVYZ8G6n8/VkRYc84fcEI/JnY/Xv_TBpMXM3U/s1600/Screen%2BShot%2B2015-11-12%2Bat%2B2.43.48%2BPM.png>
> On Thursday, November 5, 2015 at 12:57:22 AM UTC+5:30, Patrick O'Leary 
> wrote:
>>
>> It's not in the repo, it's a GitHub feature. But it may only be visible 
>> if you have Collaborator or Owner status on the repository.
>>
>> On Wednesday, November 4, 2015 at 10:51:34 AM UTC-6, Ben Ward wrote:
>>>
>>> I don't think the /graphs folder is part of the julia repo any-more :(
>>>
>>> On Wed, Nov 4, 2015 at 4:48 PM, Tony Kelman <to...@kelman.net 
>>> > wrote:
>>>
>>>> There are some interesting numbers at 
>>>> https://github.com/JuliaLang/julia/graphs/traffic
>>>>
>>>> Elliot Saba also did some scraping of the AWS download logs for 
>>>> binaries and shared the aggregate numbers (broken down by platform) 
>>>> privately with a few people, it may be worth sharing those publicly.
>>>>
>>>>
>>>> On Wednesday, November 4, 2015 at 8:26:19 AM UTC-8, Ben Ward wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I was wondering are there any metrics or stats available that show how 
>>>>> the user-base of Julia has grown over the last few years, and what it's 
>>>>> size is now?
>>>>>
>>>>> Many Thanks,
>>>>> Ben W.
>>>>>
>>>>
>>>

[julia-users] Getting length of step range in function.

[Ben Ward]

Hi,

You can get the length of a StepRange:

*julia> **length(2:1:9)*

*8*

But if I try to do so inside a function I get an error about not finding a 
call method:

*julia> **function myfun(width::Int, step::Int, length::Int)*

   *clip = Int(floor(width / 2))*

   *start = 1 + clip*

   *stop = length - clip*

   *range = StepRange(start, step, stop)*

   *return length(range)*

   *end*

*myfun (generic function with 1 method)*



*julia> **myfun(3, 1, 10)*

*ERROR: MethodError: `call` has no method matching call(::Int64, 
::StepRange{Int64,Int64})*

Closest candidates are:

  BoundsError()

  BoundsError(*::Any...*)

  DivideError()

  ...

 in myfun at none:6

I don't know why this is, the range constructed with the function call 
`myfun(3,1,10` should be the same range as 2:1:9. Why is this happening?

Thanks,
Ben.

Re: [julia-users] Re: The growth of Julia userbase

[Ben Ward]

Could anyone who can see it tell me the current figures? Or see to making
this data public? It seems an odd thing to keep private - I would presume
growth figures were something to show off.

On Wed, Nov 4, 2015 at 7:28 PM, Luthaf <lut...@luthaf.fr> wrote:

> These graphs are not available for people who are not (Github)
> collaborator for the julia repository.
>
> Tony Kelman a écrit :
>
> There are some interesting numbers at
> https://github.com/JuliaLang/julia/graphs/traffic
>
> Elliot Saba also did some scraping of the AWS download logs for binaries
> and shared the aggregate numbers (broken down by platform) privately with a
> few people, it may be worth sharing those publicly.
>
>
> On Wednesday, November 4, 2015 at 8:26:19 AM UTC-8, Ben Ward wrote:
>
> Hi all,
>
> I was wondering are there any metrics or stats available that show how the
> user-base of Julia has grown over the last few years, and what it's size is
> now?
>
> Many Thanks,
> Ben W.
>
>

[julia-users] The growth of Julia userbase

[Ben Ward]

Hi all,

I was wondering are there any metrics or stats available that show how the 
user-base of Julia has grown over the last few years, and what it's size is 
now?

Many Thanks,
Ben W.

Re: [julia-users] Re: The growth of Julia userbase

[Ben Ward]

I don't think the /graphs folder is part of the julia repo any-more :(

On Wed, Nov 4, 2015 at 4:48 PM, Tony Kelman <t...@kelman.net> wrote:

> There are some interesting numbers at
> https://github.com/JuliaLang/julia/graphs/traffic
>
> Elliot Saba also did some scraping of the AWS download logs for binaries
> and shared the aggregate numbers (broken down by platform) privately with a
> few people, it may be worth sharing those publicly.
>
>
> On Wednesday, November 4, 2015 at 8:26:19 AM UTC-8, Ben Ward wrote:
>>
>> Hi all,
>>
>> I was wondering are there any metrics or stats available that show how
>> the user-base of Julia has grown over the last few years, and what it's
>> size is now?
>>
>> Many Thanks,
>> Ben W.
>>
>

[julia-users] Combining arrays in an R enlist like manner

[Ben Ward]

Hi,

In R, if you have a list, containing two vectors that are - say - numeric, 
you can unlist them into one vector:

> a <- list(c(1,2,3,4,5), c(7,8,9,10,11))

> a

[[1]]

[1] 1 2 3 4 5



[[2]]

[1]  7  8  9 10 11



> unlist(a)

 [1]  1  2  3  4  5  7  8  9 10 11

> 


Is there a convenient way to do this with vectors in Julia? Say I have a 
Vector{Vector{Int}}:

*vec = Vector{Int}[[1,2,3,4,5], [6,7,8,9,10]]*

I can only think of creating a new vector and through some 
loopiness, filling it in. 

Thanks,
Ben.

[julia-users] Request for comments: best way to deal with type uncertainty.

[Ben Ward]

Hi Julia Users,

I'm one of the Core-Devs in the BioJulia organisation, with a background in 
evolutionary biology/genetics, and, with a few other contributors I'm 
writing Bio.jl's Phylo submodule.

The primary type of this submodule is the Phylogeny. Which is a composite 
type, used to describe a model of evolution. At the very minimum it looks 
like this:

type PhyNode
children::Vector{PhyNode}
parent::PhyNode

function PhyNode(children::Vector{PhyNode} = PhyNode[],
 parent = nothing)
x = new()
if parent != nothing
graft!(parent, x)
else
x.parent = x
end
x.children = PhyNode[]
for child in children
graft!(x, child)
end
return x
end
end

type Phylogeny
root::PhyNode
rooted::Bool
rerootable::Bool

Phylogeny() = new(PhyNode(), false, true)
end

PhyNodes are types which link to their children and to their parent - they 
are the individual objects that form the tree structure. The Phylogeny type 
describes the overall tree, and contains a variable pointing to a PhyNode 
that forms the root of the tree, and determines whether the tree is rooted 
in the phylogenetic sense, and whether the phylogeny is re-rootable. So far 
so good. We can represent the structure of a phylogeny - a model of how 
various species are related through history.

Here is where I'd like comments from the julia-users, if possible: With a 
phylogeny, often additional information is annotated to the tree, like 
branch lengths, confidence intervals, sequences, labels, colours for 
plotting, and so on. Well, we can do this with a Dict, and use PhyNodes as 
keys:

typealias NodeAnnotation{T} Dict{PhyNode, T}

We can then store thee annotations in the Phylogeny type like this:
type Phylogeny{S <: AbstractString}
root::PhyNode
rooted::Bool
rerootable::Bool
annotations::Dict{S, Any}
end

However, I don't like the type uncertainty of Any because if I'm correct, 
it could propagate up through a user's code. But we will always have some 
uncertainty, because we don't know in advance what the user might want to 
annotate the Phylogeny with - could be anything from simple float values, 
to other composite types.

Am I correct that the uncertainty getting and setting such annotations, 
would propagate through the user's code when they deal with annotations?
If so, we have tried to think of ways to get around this. One idea was to 
store the NodeAnnotations in the phylogeny according to the type of their 
values, and then provide getter and setter methods that make the return 
type predictable from the types of the parameters passed in the method:

type Phylogeny{S<:AbstractString}
root::PhyNode
rooted::Bool
rerootable::Bool
annotations::Dict{Type, Dict{S, NodeAnnotation{Any}}}
end

function setannotation!{T}(x::Phylogeny, name::ASCIIString, ann::
NodeAnnotation{T})
if haskey(x.annotations,T)
x.annotations[T][name] = ann
else 
x.annotations[T] = [name => ann]
end
end 

function getannotations{T}(x::Phylogeny, name::ASCIIString, ::Type{T})
x.annotations[T][name]::Dict{PhyNode, T}
end

This seems like it works and would indeed make getting and setting more 
type predictable, the only annoying part is that Dicts get converted:

julia> setannotation!(tree, "Node Names", NodeAnnotation{ASCIIString}())
Dict{PhyNode,ASCIIString} with 0 entries


julia> tree
Phylogeny{ASCIIString}(PhyNode(),false,false,Dict{Type{T},Dict{ASCIIString,
Dict{PhyNode,Any}}}(ASCIIString=>Dict("Node Names"=>Dict{PhyNode,Any}(

You see Dict{PhyNode, ASCIIString} got converted to Dict{PhyNode, Any}.

If anyone has comments on this or has advice on how to prevent type 
uncertainty propagating, please do share. How should we be approaching this?

Many thanks,
Ben.

[julia-users] Re: Request for comments: best way to deal with type uncertainty.

[Ben Ward]

As an update: We have tested fetching annotations without trying to enforce 
type, and then another in which we don't. I don't understand why, but the 
one in which we don't enforce type, is faster, it is also puzzling for me 
as the one where we don't enforce type allocates memory, and yet is still 
faster:

function getannotations{T}(x::Phylogeny, name::ASCIIString, ::Type{T})
x.annotations[T][name]::T 
end



function getannotations(x::Phylogeny, name::ASCIIString)
for (k, v) in x.annotations
if haskey(v, name)
return(v[name])
 end
end
error("No such key in the phylogeny")
end


*julia> **@time for i in 1:1; a = tree["Node Names", ASCIIString];**end*

  0.002090 seconds

*julia> **@time for i in 1:1; a = tree["Node Names"]; **end*

  0.001367 seconds (10.00 k allocations: 312.500 KB)





 

On Thursday, September 24, 2015 at 8:17:52 PM UTC+1, Ben Ward wrote:
>
> Hi Julia Users,
>
> I'm one of the Core-Devs in the BioJulia organisation, with a background 
> in evolutionary biology/genetics, and, with a few other contributors I'm 
> writing Bio.jl's Phylo submodule.
>
> The primary type of this submodule is the Phylogeny. Which is a composite 
> type, used to describe a model of evolution. At the very minimum it looks 
> like this:
>
> type PhyNode
> children::Vector{PhyNode}
> parent::PhyNode
> 
> function PhyNode(children::Vector{PhyNode} = PhyNode[],
>  parent = nothing)
> x = new()
> if parent != nothing
> graft!(parent, x)
> else
> x.parent = x
> end
> x.children = PhyNode[]
> for child in children
> graft!(x, child)
> end
> return x
> end
> end
>
> type Phylogeny
> root::PhyNode
> rooted::Bool
> rerootable::Bool
>
> Phylogeny() = new(PhyNode(), false, true)
> end
>
> PhyNodes are types which link to their children and to their parent - they 
> are the individual objects that form the tree structure. The Phylogeny type 
> describes the overall tree, and contains a variable pointing to a PhyNode 
> that forms the root of the tree, and determines whether the tree is rooted 
> in the phylogenetic sense, and whether the phylogeny is re-rootable. So far 
> so good. We can represent the structure of a phylogeny - a model of how 
> various species are related through history.
>
> Here is where I'd like comments from the julia-users, if possible: With a 
> phylogeny, often additional information is annotated to the tree, like 
> branch lengths, confidence intervals, sequences, labels, colours for 
> plotting, and so on. Well, we can do this with a Dict, and use PhyNodes as 
> keys:
>
> typealias NodeAnnotation{T} Dict{PhyNode, T}
>
> We can then store thee annotations in the Phylogeny type like this:
> type Phylogeny{S <: AbstractString}
> root::PhyNode
> rooted::Bool
> rerootable::Bool
> annotations::Dict{S, Any}
> end
>
> However, I don't like the type uncertainty of Any because if I'm correct, 
> it could propagate up through a user's code. But we will always have some 
> uncertainty, because we don't know in advance what the user might want to 
> annotate the Phylogeny with - could be anything from simple float values, 
> to other composite types.
>
> Am I correct that the uncertainty getting and setting such annotations, 
> would propagate through the user's code when they deal with annotations?
> If so, we have tried to think of ways to get around this. One idea was to 
> store the NodeAnnotations in the phylogeny according to the type of their 
> values, and then provide getter and setter methods that make the return 
> type predictable from the types of the parameters passed in the method:
>
> type Phylogeny{S<:AbstractString}
> root::PhyNode
> rooted::Bool
> rerootable::Bool
> annotations::Dict{Type, Dict{S, NodeAnnotation{Any}}}
> end
>
> function setannotation!{T}(x::Phylogeny, name::ASCIIString, ann::
> NodeAnnotation{T})
> if haskey(x.annotations,T)
> x.annotations[T][name] = ann
> else 
> x.annotations[T] = [name => ann]
> end
> end 
>
> function getannotations{T}(x::Phylogeny, name::ASCIIString, ::Type{T})
> x.annotations[T][name]::Dict{PhyNode, T}
> end
>
> This seems like it works and would indeed make getting and setting more 
> type predictable, the only annoying part is that Dicts get converted:
>
> julia> setannotation!(tree, "Node Names", NodeAnnotation{ASCIIString}())
> Dict{PhyNode,ASCIIString} with 0 entries
>
>
> julia> tree
> P

Re: [julia-users] Re: Creating a show method for type alias

[Ben Ward]

I did debate doing this, but went with the typealias as, the way I 
understood it, CIGARString would be a variable containing a reference, to a 
vector or CIGARS. When in my mind, where a String is thought of as the 
array of chars, so it should be with CIGARString, it is the array of 
CIGARS. 

I should mention I had failed in the past to provide my own show methods 
for arrays containing some types, which is - at the heart of the matter - 
what I'm trying to do; create my own show method for and array of CIGARs. 

I solved this problem with the addition of a write mime method. This 
solution was thanks to the wonderful D.C. Jones':

function writemime(io::IO, ::MIME{symbol(text/plain)}, cs::CIGARString)
show(io, cs)
end






On Monday, August 17, 2015 at 12:25:44 AM UTC+1, Colin Bowers wrote:

 No. Apologies. I didn't read the question closely enough.

 However, upon closer reading, I cannot see any reason why you need 
 CIGARString to be a type alias. Wouldn't it make more sense for CIGARString 
 to also be its own type?

 type CIGARString
 x::Vector{CIGAR}
 end

 I think this would solve all your problems in one hit and you would end up 
 with something that is more versatile than using a type alias... If you 
 want to treat an instance of CIGARString like you would any old vector, 
 then just overload the standard methods, e.g. getindex, setindex!, e.t.c.

 On 15 August 2015 at 09:26, Ben Ward axolotl...@gmail.com javascript: 
 wrote:

 Would that work? The variable to be printed isn't fed in as the second 
 option? 


 On Wednesday, August 12, 2015 at 1:34:16 AM UTC+1, colint...@gmail.com 
 wrote:

 Does the following work?

 function Base.show(io::IO, ::Type{CIGARString})
 #your code here
 end



 On Tuesday, 11 August 2015 03:07:15 UTC+10, Ben Ward wrote:

 Hi, I have implemented a basic immutable type with a type alias for a 
 vector of said type:

 immutable CIGAR
 OP::Operation
 Size::Int
 end


 function CIGAR(op::Char, size::Int)
 return CIGAR(Operation(op), size)
 end


 function convert(::Type{String}, cigar::CIGAR)
 return $(cigar.Size)$(Char(cigar.OP))
 end


 function show(io::IO, cigar::CIGAR)
 write(io, convert(String, cigar))
 end


 typealias CIGARString Vector{CIGAR}


 function convert(::Type{CIGARString}, str::String)
 matches = matchall(r(\d+)(\w), str)
 cigarString = Vector{CIGAR}(length(matches))
 @inbounds for i in 1:length(matches)
 m = matches[i]
 cigarString[i] = CIGAR(m[end], parse(Int, m[1:end-1]))
 end
 return cigarString
 end


 macro cigar_str(str)
 return CIGARString(str)
 end

 I also want to define a show method for the alias CIGARString, so as it 
 is converted to a string that can be used with a show method:

 function convert(::Type{String}, cigarString::CIGARString)
 outString = 
 for cigar in cigarString
 outString *= String(cigar)
 end
 return outString
 end


 function show(io::IO, cigarstr::CIGARString)
 write(io, convert(String, cigarstr))
 end

 However the output a see on the REPL is:

 *3-element Array{Bio.Align.CIGAR,1}:*

 * 5M*

 * 5N*

 * 5M*

 So, rather than the show method for CIGARString being called, the show 
 method for CIGAR is being called repeatedly for every element in the 
 CIGARString vector. How do I get Julia to use the show method I want for 
 CIGARString?

 Thanks.

[julia-users] Re: Creating a show method for type alias

[Ben Ward]

Would that work? The variable to be printed isn't fed in as the second 
option? 

On Wednesday, August 12, 2015 at 1:34:16 AM UTC+1, colint...@gmail.com 
wrote:

 Does the following work?

 function Base.show(io::IO, ::Type{CIGARString})
 #your code here
 end



 On Tuesday, 11 August 2015 03:07:15 UTC+10, Ben Ward wrote:

 Hi, I have implemented a basic immutable type with a type alias for a 
 vector of said type:

 immutable CIGAR
 OP::Operation
 Size::Int
 end


 function CIGAR(op::Char, size::Int)
 return CIGAR(Operation(op), size)
 end


 function convert(::Type{String}, cigar::CIGAR)
 return $(cigar.Size)$(Char(cigar.OP))
 end


 function show(io::IO, cigar::CIGAR)
 write(io, convert(String, cigar))
 end


 typealias CIGARString Vector{CIGAR}


 function convert(::Type{CIGARString}, str::String)
 matches = matchall(r(\d+)(\w), str)
 cigarString = Vector{CIGAR}(length(matches))
 @inbounds for i in 1:length(matches)
 m = matches[i]
 cigarString[i] = CIGAR(m[end], parse(Int, m[1:end-1]))
 end
 return cigarString
 end


 macro cigar_str(str)
 return CIGARString(str)
 end

 I also want to define a show method for the alias CIGARString, so as it 
 is converted to a string that can be used with a show method:

 function convert(::Type{String}, cigarString::CIGARString)
 outString = 
 for cigar in cigarString
 outString *= String(cigar)
 end
 return outString
 end


 function show(io::IO, cigarstr::CIGARString)
 write(io, convert(String, cigarstr))
 end

 However the output a see on the REPL is:

 *3-element Array{Bio.Align.CIGAR,1}:*

 * 5M*

 * 5N*

 * 5M*

 So, rather than the show method for CIGARString being called, the show 
 method for CIGAR is being called repeatedly for every element in the 
 CIGARString vector. How do I get Julia to use the show method I want for 
 CIGARString?

 Thanks.

Re: [julia-users] Creating a show method for type alias

[Ben Ward]

I tried this too, to override the method for Array{CIGAR, 1} but it still 
does not work:

function Base.show(io::IO, cigarstr::Array{CIGAR, 1})
write(io, convert(String, cigarstr))
end


On Tuesday, August 11, 2015 at 9:22:09 AM UTC+1, Mauro wrote:

 I think you cannot create a show method for a typealias as it is 
 indistinguishable from the type it aliases. 

 On Mon, 2015-08-10 at 19:07, Ben Ward axolotl...@gmail.com javascript: 
 wrote: 
  Hi, I have implemented a basic immutable type with a type alias for a 
 vector of 
  said type: 
  
  immutable CIGAR 
OP::Operation 
Size::Int 
  end 
  
  
  function CIGAR(op::Char, size::Int) 
return CIGAR(Operation(op), size) 
  end 
  
  
  function convert(::Type{String}, cigar::CIGAR) 
return $(cigar.Size)$(Char(cigar.OP)) 
  end 
  
  
  function show(io::IO, cigar::CIGAR) 
write(io, convert(String, cigar)) 
  end 
  
  
  typealias CIGARString Vector{CIGAR} 
  
  
  function convert(::Type{CIGARString}, str::String) 
matches = matchall(r(\d+)(\w), str) 
cigarString = Vector{CIGAR}(length(matches)) 
@inbounds for i in 1:length(matches) 
  m = matches[i] 
  cigarString[i] = CIGAR(m[end], parse(Int, m[1:end-1])) 
end 
return cigarString 
  end 
  
  
  macro cigar_str(str) 
return CIGARString(str) 
  end 
  
  I also want to define a show method for the alias CIGARString, so as it 
 is 
  converted to a string that can be used with a show method: 
  
  function convert(::Type{String}, cigarString::CIGARString) 
outString =  
for cigar in cigarString 
  outString *= String(cigar) 
end 
return outString 
  end 
  
  
  function show(io::IO, cigarstr::CIGARString) 
write(io, convert(String, cigarstr)) 
  end 
  
  However the output a see on the REPL is: 
  
  
  3-element Array{Bio.Align.CIGAR,1}: 
  
  
  5M 
  
  
  5N 
  
  
  5M 
  
  
  So, rather than the show method for CIGARString being called, the show 
 method 
  for CIGAR is being called repeatedly for every element in the 
 CIGARString 
  vector. How do I get Julia to use the show method I want for 
 CIGARString? 
  
  Thanks. 

[julia-users] Creating a show method for type alias

[Ben Ward]

Hi, I have implemented a basic immutable type with a type alias for a 
vector of said type:

immutable CIGAR
OP::Operation
Size::Int
end


function CIGAR(op::Char, size::Int)
return CIGAR(Operation(op), size)
end


function convert(::Type{String}, cigar::CIGAR)
return $(cigar.Size)$(Char(cigar.OP))
end


function show(io::IO, cigar::CIGAR)
write(io, convert(String, cigar))
end


typealias CIGARString Vector{CIGAR}


function convert(::Type{CIGARString}, str::String)
matches = matchall(r(\d+)(\w), str)
cigarString = Vector{CIGAR}(length(matches))
@inbounds for i in 1:length(matches)
m = matches[i]
cigarString[i] = CIGAR(m[end], parse(Int, m[1:end-1]))
end
return cigarString
end


macro cigar_str(str)
return CIGARString(str)
end

I also want to define a show method for the alias CIGARString, so as it is 
converted to a string that can be used with a show method:

function convert(::Type{String}, cigarString::CIGARString)
outString = 
for cigar in cigarString
outString *= String(cigar)
end
return outString
end


function show(io::IO, cigarstr::CIGARString)
write(io, convert(String, cigarstr))
end

However the output a see on the REPL is:

*3-element Array{Bio.Align.CIGAR,1}:*

* 5M*

* 5N*

* 5M*

So, rather than the show method for CIGARString being called, the show 
method for CIGAR is being called repeatedly for every element in the 
CIGARString vector. How do I get Julia to use the show method I want for 
CIGARString?

Thanks.

[julia-users] Julia's Scheduler and Event Loop

[Ben Ward]

Hi there,

I'd like to try some event driven programming, where a segment of code is 
set to run, but only when an event occurs - think like EventListeners in 
Node.js.

I think this should be pretty easy with Julia's Scheduler, and I'm 
currently going over the standard library reference documentation to see 
how this would be done. I see that notify() and conditions are rather like 
events. I suppose if you schedule a task that waits for a condition as the 
first statement then it functions like an event listener. 
I guess if you then put the code in an infinite loop with the statement to 
wait on the conditions, this would be like the event listeners that are 
fireweed repeatedly whenerve the event is fired, whereas otherwise the task 
would just be completed and removed from the scheduler when it was 
finished, and so is more like the 'once' event listeners from Node.js.

I wonder however if there is more detailed information out there on how the 
Scheduler works - for example, in the flow of a 'normal script', when 
exactly does control pass to the scheduler: I guess for @schedule, flow 
goes to the task right away, but say you had several tasks scheduled to be 
run in a script with several @schedule or @task statements - not waiting 
for a condition. Does the control go from the script to the task which 
finishes, back to the script to add the second task to the scheduler, which 
gets finished. Or once control is switched to the scheduler, does it work 
through all the tasks it has sued until they are all waiting on something, 
and then at that point control passes back to the main script? In addition, 
what variables from my script are accessible from within a task?

Thanks,
Ben.

[julia-users] Getting Errors / messages from an external program.

[Ben Ward]

Hi,

I want to run an external command in Julia. Should a command fail I want to 
get the message from the external command.

I know failed commands raise a general exception that can be caught:

*run(`dat hi`)*

dat: hi is not a valid command

*ERROR: failed process: Process(`dat hi`, ProcessExited(1)) [1]*

* in run at ./process.jl:490*

I see that a message is printed out by the external program:

dat: hi is not a valid command.

However readall() does not get this message and return it as a string:

*readall(`dat hi`)*

dat: hi is not a valid command

*ERROR: failed process: Process(`dat hi`, ProcessExited(1)) [1]*

* in readbytes at process.jl:474*

* in readall at process.jl:479*

Is there a way I can capture this message at all and report it back to the 
user as part of a try-catch upon catching of the ProcessExited(1) exception?

Thanks,
Ben.

[julia-users] Re: Creating function with a macro

[Ben Ward]

For the toy example you are right, parametric function would be better - 
what I'm actually trying to do is define several `searchsortfirst` 
functions, where the lt function is different. 

This is because I have a vector of one immutable composite type of several 
values I wish to sort and search. I could sort/search them by their first 
value, second value, or third value, and there is no canonical lt for the 
type. 

Base.sort can be provided an anonymous function to use as lt - but 
performance is absolutely critical in my use case and I need to squeeze as 
much performance as I can. Benchmark's I have done have shown that using 
anonymous functions slow things down. However, if I copy the searchsorted 
code and modify it to contain my custom lt condition, then everything is 
faster.

Therefore I wanted to make a macro that would define several of my custom 
searchsortedfirst functions: If I provided it a name for the function, the 
types/arguments is accepts, and then the custom lt expression. It would 
return a definition of the custom function with the lt condition hard coded 
in.


On Wednesday, June 17, 2015 at 8:07:11 PM UTC+1, Tom Breloff wrote:

 My gut reaction is that you don't want to use a macro here.  Can you use a 
 parametric definition:
 f{T}(vectype:T) = do something useful with the T

 or can you just use multiple dispatch:
 f{T:FloatingPoint}(v::Vector{T}) = something for floats
 f{T:Integer}(v::Vector{T}) = something for ints

 What's your use case?

 But to answer your question... I think this should work:


 julia macro customFun(vectype::Expr, name::Symbol)
quote
function $(esc(name))(v::$(esc(vectype)))
println(typeof(v))
end
end
end

 julia @customFun Vector{Int} f
 f (generic function with 1 method)

 julia f(Int[])
 Array{Int64,1}




 On Wednesday, June 17, 2015 at 2:13:41 PM UTC-4, Ben Ward wrote:

 Hi, I want to create a macro with which I can create a function with a 
 custom bit of code:

 In the repl I can do a toy example:


 *name = :hi*


 *vectype = Vector{Int}*


 *quote**  function ($name)(v::$vectype)*

 *println(hi)*

   *end*

 *end*

 However if I try to put this in a macro and use it I get an error:

 *macro customFun(vectype::DataType, name::Symbol)*


 *  quote**   function ($name)(v::$vectype)*

 *  println(hi World!)*

 *end*

 *  end*



 *end*

 *@customFun(Vector{Int}, :hi)*

 What am I doing wrong? I'd like to use macro arguments to provide a 
 function's name, and the datatype of the argument. I haven't used macros to 
 define functions more complex than simple one liners.

 Thanks,
 Ben.

[julia-users] Creating function with a macro

[Ben Ward]

Hi, I want to create a macro with which I can create a function with a 
custom bit of code:

In the repl I can do a toy example:


*name = :hi*


*vectype = Vector{Int}*


*quote**  function ($name)(v::$vectype)*

*println(hi)*

  *end*

*end*

However if I try to put this in a macro and use it I get an error:

*macro customFun(vectype::DataType, name::Symbol)*


*  quote**   function ($name)(v::$vectype)*

*  println(hi World!)*

*end*

*  end*



*end*

*@customFun(Vector{Int}, :hi)*

What am I doing wrong? I'd like to use macro arguments to provide a 
function's name, and the datatype of the argument. I haven't used macros to 
define functions more complex than simple one liners.

Thanks,
Ben.

[julia-users] Re: Creating function with a macro

[Ben Ward]

I guess for Base sort! passing a functior overriding call() as lt is 
possible, if it accepts the three arguments passed to lt in lt(o, x, v[j-1])

On Wednesday, June 17, 2015 at 9:11:22 PM UTC+1, Kristoffer Carlsson wrote:

 You could also use something called functors which basically are types 
 that overload the call function. When you pass these as argument the 
 compiler can specialize the function on the type of the functor and thus 
 inline the call. See here for example for them being used effectively for 
 performance increase: https://github.com/JuliaLang/julia/pull/11685

 As an example I took the code for insertionsort and made it instead accept 
 an argument f which will be the functor. I then create some functors to 
 sort on the different type fields and show an example how sort is called.


 function sort!(v::AbstractVector, f, lo::Int=1, hi::Int=length(v))
 @inbounds for i = lo+1:hi
 j = i
 x = v[i]
 while j  lo
 if f(x, v[j-1])
 v[j] = v[j-1]
 j -= 1
 continue
 end
 break
 end
 v[j] = x
 end
 return v
 end

 # Some type
 immutable CompType
 a::Int
 b::Int
 c::Int
 end


 b = [CompType(1,2,3), CompType(3,2,1), CompType(2,1,3)]

 # Functors 
 immutable AFunc end
 call(::AFunc, x, y) = x.a  y.a
 immutable BFunc end
 call(::BFunc, x, y) = x.b  y.b
 immutable CFunc end
 call(::CFunc, x, y) = x.c  y.c

 # Can now sort with good performance
 sort!(b, AFunc())
 println(b)
 sort!(b, BFunc())
 println(b)
 sort!(b, CFunc())
 println(b)


 Now, this is of course not optimal to rip code out of base. It would be 
 better if we could pass a functor straight to Base.sort!. 


 On Wednesday, June 17, 2015 at 8:13:41 PM UTC+2, Ben Ward wrote:

 Hi, I want to create a macro with which I can create a function with a 
 custom bit of code:

 In the repl I can do a toy example:


 *name = :hi*


 *vectype = Vector{Int}*


 *quote**  function ($name)(v::$vectype)*

 *println(hi)*

   *end*

 *end*

 However if I try to put this in a macro and use it I get an error:

 *macro customFun(vectype::DataType, name::Symbol)*


 *  quote**   function ($name)(v::$vectype)*

 *  println(hi World!)*

 *end*

 *  end*



 *end*

 *@customFun(Vector{Int}, :hi)*

 What am I doing wrong? I'd like to use macro arguments to provide a 
 function's name, and the datatype of the argument. I haven't used macros to 
 define functions more complex than simple one liners.

 Thanks,
 Ben.

[julia-users] Re: Creating function with a macro

[Ben Ward]

Actually, looking at it again with fresh eyes - I can define a few new 
classes inhering from Ordering, and then define my own lt(o, x, [j-1]) for 
each. It looks like these predefined orderings and lt methods in the 
sorting api - ForwardOrdering, ReverseOrdering and so on are fast as their 
lt() methods are defined as:

lt(o::ForwardOrdering,   a, b) = isless(a,b)lt(o::ReverseOrdering, 
  a, b) = lt(o.fwd,b,a)

Whereas with feeding in custom functions it builds a Lt Operdering type 
that contains a reference for the custom Lt function - which I believe is 
the slow version.

On Wednesday, June 17, 2015 at 11:20:47 PM UTC+1, Ben Ward wrote:

 I guess for Base sort! passing a functior overriding call() as lt is 
 possible, if it accepts the three arguments passed to lt in lt(o, x, v[j-1
 ])

 On Wednesday, June 17, 2015 at 9:11:22 PM UTC+1, Kristoffer Carlsson wrote:

 You could also use something called functors which basically are types 
 that overload the call function. When you pass these as argument the 
 compiler can specialize the function on the type of the functor and thus 
 inline the call. See here for example for them being used effectively for 
 performance increase: https://github.com/JuliaLang/julia/pull/11685

 As an example I took the code for insertionsort and made it instead 
 accept an argument f which will be the functor. I then create some functors 
 to sort on the different type fields and show an example how sort is called.


 function sort!(v::AbstractVector, f, lo::Int=1, hi::Int=length(v))
 @inbounds for i = lo+1:hi
 j = i
 x = v[i]
 while j  lo
 if f(x, v[j-1])
 v[j] = v[j-1]
 j -= 1
 continue
 end
 break
 end
 v[j] = x
 end
 return v
 end

 # Some type
 immutable CompType
 a::Int
 b::Int
 c::Int
 end


 b = [CompType(1,2,3), CompType(3,2,1), CompType(2,1,3)]

 # Functors 
 immutable AFunc end
 call(::AFunc, x, y) = x.a  y.a
 immutable BFunc end
 call(::BFunc, x, y) = x.b  y.b
 immutable CFunc end
 call(::CFunc, x, y) = x.c  y.c

 # Can now sort with good performance
 sort!(b, AFunc())
 println(b)
 sort!(b, BFunc())
 println(b)
 sort!(b, CFunc())
 println(b)


 Now, this is of course not optimal to rip code out of base. It would be 
 better if we could pass a functor straight to Base.sort!. 


 On Wednesday, June 17, 2015 at 8:13:41 PM UTC+2, Ben Ward wrote:

 Hi, I want to create a macro with which I can create a function with a 
 custom bit of code:

 In the repl I can do a toy example:


 *name = :hi*


 *vectype = Vector{Int}*


 *quote**  function ($name)(v::$vectype)*

 *println(hi)*

   *end*

 *end*

 However if I try to put this in a macro and use it I get an error:

 *macro customFun(vectype::DataType, name::Symbol)*


 *  quote**   function ($name)(v::$vectype)*

 *  println(hi World!)*

 *end*

 *  end*



 *end*

 *@customFun(Vector{Int}, :hi)*

 What am I doing wrong? I'd like to use macro arguments to provide a 
 function's name, and the datatype of the argument. I haven't used macros to 
 define functions more complex than simple one liners.

 Thanks,
 Ben.

Re: [julia-users] Re: upper_bound and lower_bound equivalents for Julia

[Ben Ward]

Just gave it a go - My code translated from C++ still performs very well.


function runss(arr, limits) 
  s = 0 
  for l in limits 
ind = searchsorted(arr, true, by = x-xl) 
s += first(ind) 
  end 
  s 
end 

function runff(arr, limits) 
  s = 0 
  for l in limits 
ind = findfirst(arr . l) 
s += ind
  end 
  s 
end 

function runup(arr, limits) 
  s = 0 
  for l in limits 
ind = upperBound(arr, 1, length(arr), l) 
s += ind 
  end 
  s 
end 

@time runup(arr, unique(arr))

16.235 microseconds (20 allocations: 944 bytes)

*19*
@time runff(arr, unique(arr))

4.569 milliseconds (1860 allocations: 104 KB)

*11*
@time runss(arr, unique(arr))

30.018 microseconds (35 allocations: 1472 bytes)

*3*





On Saturday, June 13, 2015 at 11:20:06 AM UTC+1, Tim Holy wrote:

 You can also use searchsorted (though because it returns a range, it has 
 twice 
 as many function evaluations as your upperBound): 

 searchsorted(arr, true, by = x-x10) 

 As a small point (because your overall conclusion is of course correct), 
 if 
 you want to evaluate performance, you usually also want at least 
 milliseconds 
 of execution time (and make sure to run things twice). Here's a good test 
 script: 

 function runss(arr, limits) 
s = 0 
for l in limits 
ind = searchsorted(arr, true, by = x-xl) 
s += first(ind) 
end 
s 
end 

 @time runss(arr, linspace(0.1,0.9,1)) 

 and a similar function for any alternatives. By returning s, you ensure 
 that 
 LLVM won't do anything tricky like noticing you don't actually use the 
 output 
 value and optimize your test away. 

 --Tim 


 On Friday, June 12, 2015 08:52:56 PM Ben Ward wrote: 
  Just to update, I realised this might be a prime candidate for 
 findfirst. 
  It can imitate the behaviour of upper_bound - returning the idx of the 
  first array element which 
  
  
  
  *arr = [10, 10, 10, 20, 20, 20, 30, 30]* 
  
  
  *findfirst(x - x  10, arr)4* 
  
  However, if I quickly scribble down the code from the C++ page linked to 
  earlier - which is supposed to demonstrate the upper_bound behaviour, I 
 get 
  a speed boost: 
  
  *function upperBound(array, first, last, value, comp = x - value  x)* 
  
*  count = last - first* 
  
*  idx = 0* 
  
*  while count  0* 
  
*idx = first* 
  
*step = Int(round(count / 2))* 
  
*idx += step* 
  
*if !comp(array[idx])* 
  
*  idx += 1* 
  
*  first = idx* 
  
*  count -= step + 1* 
  
*else* 
  
*  count = step* 
  
*end* 
  
*  end* 
  
*  return first* 
  
  
  
  *end* 
  
  *@time findfirst(x - x  10, arr)* 
  
1.606 milliseconds (25 allocations: 1642 bytes) 
  
  
  
  
  
  *4* 
  
  *@time upperBound(arr, 1, length(arr), 10)* 
  
7.284 microseconds (6 allocations: 208 bytes) 
  
  *4* 
  
  Thanks, 
  Ben. 
  
  On Saturday, June 13, 2015 at 2:41:27 AM UTC+1, Ben Ward wrote: 
   Hi, 
   
   I'm implementing some C++ code in Julia and the code uses the 
 upper_bound 
   and lower_bound functions from the C++ standard library 
   http://www.cplusplus.com/reference/algorithm/lower_bound/. 
   
   I was wondering if a method for achieving a similar result with arrays 
 has 
   been achieved in Julia? 
   
   Thanks, 
   Ben. 

Re: [julia-users] Re: upper_bound and lower_bound equivalents for Julia

[Ben Ward]

Thanks Kevin,

I hadn't seen searchsortedfirst until now, presumably you can specify a 
function to use to sort more complex objects (for example an array of 
immutables with two numeric fields and you want to sort and match based on 
one of those fields)

Thanks,
Ben.

On Saturday, June 13, 2015 at 6:07:51 PM UTC+1, Kevin Squire wrote:

 Some quick comments:

 1. `searchsorted(arr, true, by= x- xl)` is equivalent to 
 `searchsorted(arr, l)`.  Making this replacement gives me nearly identical 
 timing to upperBound.
 2. `searchsortedfirst(arr, l)` is actually what you want, as it only 
 returns the first element greater than a given bound
 3. `searchsorted` doesn't actually do twice as much work. 
 https://github.com/JuliaLang/julia/blob/master/base/sort.jl#L163-L179, 
 although how much extra it does do depends on the length of the sequence of 
 identical values you're searching for. 

 Cheers!
Kevin

 On Sat, Jun 13, 2015 at 8:21 AM, Tim Holy tim@gmail.com javascript:
  wrote:

 Yep, and the advantage compared to findfirst will only increase with 
 larger
 arrays (try arr = sort(rand(10^7)) to see what I mean), because findfirst 
 is
 O(N) whereas the other two are O(logN). searchsorted and upperBound
 essentially use the same algorithm, but searchsorted does twice as much 
 work
 (finds the first  last, not just the first), so you'll always see a 
 factor of 2
 advantage for upperBound.

 Another great example of the power of being able to write algorithms 
 tuned to
 the problem you care about, rather than being forced to use library 
 functions.

 --Tim

 On Saturday, June 13, 2015 07:07:22 AM Ben Ward wrote:
  Just gave it a go - My code translated from C++ still performs very 
 well.
 
 
  function runss(arr, limits)
s = 0
for l in limits
  ind = searchsorted(arr, true, by = x-xl)
  s += first(ind)
end
s
  end
 
  function runff(arr, limits)
s = 0
for l in limits
  ind = findfirst(arr . l)
  s += ind
end
s
  end
 
  function runup(arr, limits)
s = 0
for l in limits
  ind = upperBound(arr, 1, length(arr), l)
  s += ind
end
s
  end
 
  @time runup(arr, unique(arr))
 
  16.235 microseconds (20 allocations: 944 bytes)
 
  *19*
  @time runff(arr, unique(arr))
 
  4.569 milliseconds (1860 allocations: 104 KB)
 
  *11*
  @time runss(arr, unique(arr))
 
  30.018 microseconds (35 allocations: 1472 bytes)
 
  *3*
 
  On Saturday, June 13, 2015 at 11:20:06 AM UTC+1, Tim Holy wrote:
   You can also use searchsorted (though because it returns a range, it 
 has
   twice
   as many function evaluations as your upperBound):
  
   searchsorted(arr, true, by = x-x10)
  
   As a small point (because your overall conclusion is of course 
 correct),
   if
   you want to evaluate performance, you usually also want at least
   milliseconds
   of execution time (and make sure to run things twice). Here's a good 
 test
   script:
  
   function runss(arr, limits)
  
  s = 0
  for l in limits
  
  ind = searchsorted(arr, true, by = x-xl)
  s += first(ind)
  
  end
  s
  
  end
  
   @time runss(arr, linspace(0.1,0.9,1))
  
   and a similar function for any alternatives. By returning s, you 
 ensure
   that
   LLVM won't do anything tricky like noticing you don't actually use the
   output
   value and optimize your test away.
  
   --Tim
  
   On Friday, June 12, 2015 08:52:56 PM Ben Ward wrote:
Just to update, I realised this might be a prime candidate for
  
   findfirst.
  
It can imitate the behaviour of upper_bound - returning the idx of 
 the
first array element which
   
   
   
*arr = [10, 10, 10, 20, 20, 20, 30, 30]*
   
   
*findfirst(x - x  10, arr)4*
   
However, if I quickly scribble down the code from the C++ page 
 linked to
earlier - which is supposed to demonstrate the upper_bound 
 behaviour, I
  
   get
  
a speed boost:
   
*function upperBound(array, first, last, value, comp = x - value  
 x)*
   
  *  count = last - first*
   
  *  idx = 0*
   
  *  while count  0*
   
  *idx = first*
   
  *step = Int(round(count / 2))*
   
  *idx += step*
   
  *if !comp(array[idx])*
   
  *  idx += 1*
   
  *  first = idx*
   
  *  count -= step + 1*
   
  *else*
   
  *  count = step*
   
  *end*
   
  *  end*
   
  *  return first*
   
*end*
   
*@time findfirst(x - x  10, arr)*
   
  1.606 milliseconds (25 allocations: 1642 bytes)
   
*4*
   
*@time upperBound(arr, 1, length(arr), 10)*
   
  7.284 microseconds (6 allocations: 208 bytes)
   
*4*
   
Thanks,
Ben.
   
On Saturday, June 13, 2015 at 2:41:27 AM UTC+1, Ben Ward wrote:
 Hi,

 I'm implementing some C++ code in Julia and the code uses

Re: [julia-users] Re: upper_bound and lower_bound equivalents for Julia

[Ben Ward]

So if I have a type:

*immutable foo*

*x::Int*

   *y::Int*

   *end*



On Saturday, June 13, 2015 at 8:17:43 PM UTC+1, Kevin Squire wrote:

 Hi Ben,

 For complex objects, you can pass in a comparison function (as with 
 searchsorted), but you'll run into the same slowdown--the comparison 
 function won't be inlined.  

 If there is a canonical ordering for the objects, you can define 
 `isless(a::MyObject, b::MyObject)`, and that should work fast.  Otherwise, 
 a custom function is your best bet.

 Note that this problem will be solved better in some future version of 
 Julia.  See, for example, https://github.com/JuliaLang/julia/pull/11685, 
 which speeds up a bunch of other functions using functors.  These ideas 
 should probably be applied to sorting as well.

 Cheers,
Kevin

 On Sat, Jun 13, 2015 at 10:35 AM, Ben Ward axolotl...@gmail.com 
 javascript: wrote:

 Thanks Kevin,

 I hadn't seen searchsortedfirst until now, presumably you can specify a 
 function to use to sort more complex objects (for example an array of 
 immutables with two numeric fields and you want to sort and match based on 
 one of those fields)

 Thanks,
 Ben.

 On Saturday, June 13, 2015 at 6:07:51 PM UTC+1, Kevin Squire wrote:

 Some quick comments:

 1. `searchsorted(arr, true, by= x- xl)` is equivalent to 
 `searchsorted(arr, l)`.  Making this replacement gives me nearly identical 
 timing to upperBound.
 2. `searchsortedfirst(arr, l)` is actually what you want, as it only 
 returns the first element greater than a given bound
 3. `searchsorted` doesn't actually do twice as much work. 
 https://github.com/JuliaLang/julia/blob/master/base/sort.jl#L163-L179, 
 although how much extra it does do depends on the length of the sequence of 
 identical values you're searching for. 

 Cheers!
Kevin

 On Sat, Jun 13, 2015 at 8:21 AM, Tim Holy tim@gmail.com wrote:

 Yep, and the advantage compared to findfirst will only increase with 
 larger
 arrays (try arr = sort(rand(10^7)) to see what I mean), because 
 findfirst is
 O(N) whereas the other two are O(logN). searchsorted and upperBound
 essentially use the same algorithm, but searchsorted does twice as much 
 work
 (finds the first  last, not just the first), so you'll always see a 
 factor of 2
 advantage for upperBound.

 Another great example of the power of being able to write algorithms 
 tuned to
 the problem you care about, rather than being forced to use library 
 functions.

 --Tim

 On Saturday, June 13, 2015 07:07:22 AM Ben Ward wrote:
  Just gave it a go - My code translated from C++ still performs very 
 well.
 
 
  function runss(arr, limits)
s = 0
for l in limits
  ind = searchsorted(arr, true, by = x-xl)
  s += first(ind)
end
s
  end
 
  function runff(arr, limits)
s = 0
for l in limits
  ind = findfirst(arr . l)
  s += ind
end
s
  end
 
  function runup(arr, limits)
s = 0
for l in limits
  ind = upperBound(arr, 1, length(arr), l)
  s += ind
end
s
  end
 
  @time runup(arr, unique(arr))
 
  16.235 microseconds (20 allocations: 944 bytes)
 
  *19*
  @time runff(arr, unique(arr))
 
  4.569 milliseconds (1860 allocations: 104 KB)
 
  *11*
  @time runss(arr, unique(arr))
 
  30.018 microseconds (35 allocations: 1472 bytes)
 
  *3*
 
  On Saturday, June 13, 2015 at 11:20:06 AM UTC+1, Tim Holy wrote:
   You can also use searchsorted (though because it returns a range, 
 it has
   twice
   as many function evaluations as your upperBound):
  
   searchsorted(arr, true, by = x-x10)
  
   As a small point (because your overall conclusion is of course 
 correct),
   if
   you want to evaluate performance, you usually also want at least
   milliseconds
   of execution time (and make sure to run things twice). Here's a 
 good test
   script:
  
   function runss(arr, limits)
  
  s = 0
  for l in limits
  
  ind = searchsorted(arr, true, by = x-xl)
  s += first(ind)
  
  end
  s
  
  end
  
   @time runss(arr, linspace(0.1,0.9,1))
  
   and a similar function for any alternatives. By returning s, you 
 ensure
   that
   LLVM won't do anything tricky like noticing you don't actually use 
 the
   output
   value and optimize your test away.
  
   --Tim
  
   On Friday, June 12, 2015 08:52:56 PM Ben Ward wrote:
Just to update, I realised this might be a prime candidate for
  
   findfirst.
  
It can imitate the behaviour of upper_bound - returning the idx 
 of the
first array element which
   
   
   
*arr = [10, 10, 10, 20, 20, 20, 30, 30]*
   
   
*findfirst(x - x  10, arr)4*
   
However, if I quickly scribble down the code from the C++ page 
 linked to
earlier - which is supposed to demonstrate the upper_bound 
 behaviour, I
  
   get
  
a speed boost:
   
*function upperBound(array, first, last, value, comp = x - value 
  x)*
   
  *  count = last

Re: [julia-users] Re: upper_bound and lower_bound equivalents for Julia

[Ben Ward]

Hi Kevin,

To take this out of the realm of hypothetical's: Specifically I have this 
type:

immutable GapAnchor
  gapPos::Int
  seqPos::Int
  function GapAnchor(gp = 0, sp = 0)
return new(gp, sp)
  end
end

and it has the following operators defined:

function Base.copy(src::GapAnchor)
  return GapAnchor(src.gapPos, src.seqPos)
end

function Base.==(a::GapAnnchor, b::GapAnchor)
  return a.gapPos == b.gapPos  a.seqPos == b.seqPos
end

function Base.!=(a::GapAnchor, b::GapAnchor)
  return !(a == b)
end

function Base.(a::GapAnchor, b::GapAnchor)
  return a.gapPos  b.gapPos || a.seqPos  b.seqPos
end

function Base.(a::GapAnchor, b::GapAnchor)
  return a.gapPos  b.gapPos || a.seqPos  b.seqPos
end

function Base.=(a::GapAnchor, b::GapAnchor)
  return a.gapPos = b.gapPos || a.seqPos  b.seqPos
end

function Base.=(a::GapAnchor, b::GapAnchor)
  return a.gapPos = b.gapPos || a.seqPos  b.seqPos
end

Notice for any one GapAnchor to be considered less than another - both 
fields must be less in one than the other, according to the Base methods 
defined for the type.

However I want to be able to do the sorting/searching according to either 
the gapPos field or the seqPos field.






*@time searchsortedfirst(arr2, x = true, by=y-y.xx)*

*ERROR: MethodError: `searchsortedfirst` has no method matching 
searchsortedfirst(::Array{boop,1})*

Closest candidates are:

 searchsortedfirst(::AbstractArray{T,1}, *::Any*, *::Int64*, *::Int64*, 
*::Base.Order.Ordering*)

 searchsortedfirst(::AbstractArray{T,1}, *::Any*, *::Base.Order.Ordering*)

 searchsortedfirst(::AbstractArray{T,1}, *::Any*)

This is the original problem that prompted my question - I have arrays of 
custom immutable type with two Int fields i.e. each foo has an x and a y, I 
want to be able to sort them such that I can find the first one greater 
than 'val' (std::upper_bound) or search for the first which is not less 
than 'val' (std::lower_bound), and I'd like to pick whether this is done 
according to the x of foo or the y of foo (in the code chunk above I chose 
to search/sort according to the x field of the foos: `x-y.xx`). 

I want to get as much performance at this task as possible, fitting in with 
Julia's provisions for sorting operations. But I wonder if - since you say 
feeding in a custom function is a slowdown if hardcoding a set of 
upper_bound and a lower_bound methods specifically for this type (something 
like a upper_bound_x, upper_bound_y, lower_bound_x, and lower_bound_y) 
would be better - I doubt it though. 

To answer your question about isles, The foo type does have the following 
operators defined:






On Saturday, June 13, 2015 at 8:17:43 PM UTC+1, Kevin Squire wrote:

 Hi Ben,

 For complex objects, you can pass in a comparison function (as with 
 searchsorted), but you'll run into the same slowdown--the comparison 
 function won't be inlined.  

 If there is a canonical ordering for the objects, you can define 
 `isless(a::MyObject, b::MyObject)`, and that should work fast.  Otherwise, 
 a custom function is your best bet.

 Note that this problem will be solved better in some future version of 
 Julia.  See, for example, https://github.com/JuliaLang/julia/pull/11685, 
 which speeds up a bunch of other functions using functors.  These ideas 
 should probably be applied to sorting as well.

 Cheers,
Kevin

 On Sat, Jun 13, 2015 at 10:35 AM, Ben Ward axolotl...@gmail.com 
 javascript: wrote:

 Thanks Kevin,

 I hadn't seen searchsortedfirst until now, presumably you can specify a 
 function to use to sort more complex objects (for example an array of 
 immutables with two numeric fields and you want to sort and match based on 
 one of those fields)

 Thanks,
 Ben.

 On Saturday, June 13, 2015 at 6:07:51 PM UTC+1, Kevin Squire wrote:

 Some quick comments:

 1. `searchsorted(arr, true, by= x- xl)` is equivalent to 
 `searchsorted(arr, l)`.  Making this replacement gives me nearly identical 
 timing to upperBound.
 2. `searchsortedfirst(arr, l)` is actually what you want, as it only 
 returns the first element greater than a given bound
 3. `searchsorted` doesn't actually do twice as much work. 
 https://github.com/JuliaLang/julia/blob/master/base/sort.jl#L163-L179, 
 although how much extra it does do depends on the length of the sequence of 
 identical values you're searching for. 

 Cheers!
Kevin

 On Sat, Jun 13, 2015 at 8:21 AM, Tim Holy tim@gmail.com wrote:

 Yep, and the advantage compared to findfirst will only increase with 
 larger
 arrays (try arr = sort(rand(10^7)) to see what I mean), because 
 findfirst is
 O(N) whereas the other two are O(logN). searchsorted and upperBound
 essentially use the same algorithm, but searchsorted does twice as much 
 work
 (finds the first  last, not just the first), so you'll always see a 
 factor of 2
 advantage for upperBound.

 Another great example of the power of being able to write algorithms 
 tuned to
 the problem you care about, rather than being forced to use

[julia-users] Re: upper_bound and lower_bound equivalents for Julia

[Ben Ward]

Just to update, I realised this might be a prime candidate for findfirst. 
It can imitate the behaviour of upper_bound - returning the idx of the 
first array element which 



*arr = [10, 10, 10, 20, 20, 20, 30, 30]*


*findfirst(x - x  10, arr)4*

However, if I quickly scribble down the code from the C++ page linked to 
earlier - which is supposed to demonstrate the upper_bound behaviour, I get 
a speed boost:

*function upperBound(array, first, last, value, comp = x - value  x)*

  *  count = last - first*

  *  idx = 0*

  *  while count  0*

  *idx = first*

  *step = Int(round(count / 2))*

  *idx += step*

  *if !comp(array[idx])*

  *  idx += 1*

  *  first = idx*

  *  count -= step + 1*

  *else*

  *  count = step*

  *end*

  *  end*

  *  return first*

  

*end*

*@time findfirst(x - x  10, arr)*

  1.606 milliseconds (25 allocations: 1642 bytes)





*4*

*@time upperBound(arr, 1, length(arr), 10)*

  7.284 microseconds (6 allocations: 208 bytes)

*4*

Thanks,
Ben.

On Saturday, June 13, 2015 at 2:41:27 AM UTC+1, Ben Ward wrote:

 Hi,

 I'm implementing some C++ code in Julia and the code uses the upper_bound 
 and lower_bound functions from the C++ standard library 
 http://www.cplusplus.com/reference/algorithm/lower_bound/.

 I was wondering if a method for achieving a similar result with arrays has 
 been achieved in Julia?

 Thanks,
 Ben.

[julia-users] upper_bound and lower_bound equivalents for Julia

[Ben Ward]

Hi,

I'm implementing some C++ code in Julia and the code uses the upper_bound 
and lower_bound functions from the C++ standard library 
http://www.cplusplus.com/reference/algorithm/lower_bound/.

I was wondering if a method for achieving a similar result with arrays has 
been achieved in Julia?

Thanks,
Ben.

[julia-users] Copying a simple immutable types

[Ben Ward]

I've written a simple immutable type and I'm a little confused by some 
behaviour I've seen:

immutable GapAnchor
  gapPos::Int 
  seqPos::Int 
end 

function Base.copy(src::GapAnchor) 
  return GapAnchor(src.gapPos, src.seqPos) 
end 

a = GapAnchor(5, 9) 

GapAnchor(5,9) 

b = copy(a) 

GapAnchor(5,9) 

a == b 

true 

a === b 

true 

is(a, b) 

true


I'm a bit confused as to why '===' would return true for a and b when it is 
supposed to check if two variables point to the same thing in memory, and 
copy constructs a new GapAnchor. 

[julia-users] Re: Copying a simple immutable types

[Ben Ward]

Ok, but they are indeed two separate instances of GapAnchor as indicated by 
the pointer?

On Sunday, April 26, 2015 at 4:12:01 PM UTC+1, Kristoffer Carlsson wrote:

 An immutable is *only* defined by its values. This works just the same as 
 say floats:

 julia b = 156.123

 156.123

 julia a = 156.123
 156.123

 julia pointer_from_objref(b)
 Ptr{Void} @0x06e73a70

 julia pointer_from_objref(a)
 Ptr{Void} @0x06e741e0

 julia a === b
 true



 On Sunday, April 26, 2015 at 4:48:23 PM UTC+2, Ben Ward wrote:

 I've written a simple immutable type and I'm a little confused by some 
 behaviour I've seen:

 immutable GapAnchor
   gapPos::Int 
   seqPos::Int 
 end 

 function Base.copy(src::GapAnchor) 
   return GapAnchor(src.gapPos, src.seqPos) 
 end 

 a = GapAnchor(5, 9) 

 GapAnchor(5,9) 

 b = copy(a) 

 GapAnchor(5,9) 

 a == b 

 true 

 a === b 

 true 

 is(a, b) 

 true


 I'm a bit confused as to why '===' would return true for a and b when it 
 is supposed to check if two variables point to the same thing in memory, 
 and copy constructs a new GapAnchor. 

[julia-users] Static parameter T doesen't occur in signature?

[Ben Ward]

Hi, I'm trying to write something that makes a type alias of Dict{PhyNode, 
T}, and then use that type alias in a composite type TreeAnnotations{T} 
as in the code below.

However Julia tells me about the TreeAnnotations{T}(x::Phylogeny) function:

Warning: static parameter T does not occur in signature for call at none:9.

The method will not be callable.


I'm not sure why - probably being dumb - but I can't tell what the issue 
is, you have to do NodeAnnotations{String}() say to make a Dict{PhyNode, 
String}


So I don't see why using T as I have is a problem.
Thanks,
Ben.

typealias NodeAnnotations{T} Dict{PhyNode, T}


type TreeAnnotations{T}
  x::Phylogeny
  annotations::NodeAnnotations{T}
end


function TreeAnnotations{T}(x::Phylogeny)
  return TreeAnnotations(x, NodeAnnotations{T}())
end


function Base.getindex{T}(x::TreeAnnotations{T}, index::PhyNode)
  return x.annotations[index]
end


function Base.setindex!{T}(x::TreeAnnotations{T}, item::T, clade::PhyNode)
  x.annotations[clade] = item
end

Re: [julia-users] Static parameter T doesen't occur in signature?

[Ben Ward]

Oh yes! T needs to be in the tuple of function arguments. Phylogeny however 
does not have any parametric fields, so I'm unsure about using T with it. 
Would:

function TreeAnnotations{T}(x::Phylogeny, ::Type{T})
  return TreeAnnotations(x, NodeAnnotations{T}())
end

Be more acceptable and suitably Julian?

Best,
Ben.

On Thursday, March 5, 2015 at 8:42:27 PM UTC, Tim Holy wrote:

 On Thursday, March 05, 2015 12:07:21 PM Ben Ward wrote: 
  However Julia tells me about the TreeAnnotations{T}(x::Phylogeny) 
 function: 
  
  Warning: static parameter T does not occur in signature for call at 
 none:9. 
  
  The method will not be callable. 

 Julia is warning you that you haven't actually used T: you've prepared 
 julia 
 for the fact that T will be a type parameter, but none of the arguments 
 are 
 parametric in T. Declare the function signature like this: 

 function TreeAnnotations{T}(x::Phylogeny{T}) 
   return TreeAnnotations(x, NodeAnnotations{T}()) 
 end 

 Notice I added {T} to the end of Phylogeny. 

 --Tim 

[julia-users] Re: Making optional fields of unknown type more efficient

[Ben Ward]

I've thought about it a bit, and I wonder if this is a better solution 
instead of expecting nodes of the tree structure to be able to work with 
many different possible types, rather using dictionaries that pair a set of 
nodes with one type of data are better:

type TreeAnnotations{T}
 x::Phylogeny
 dict::Dict{PhyNode, T}
end

So if you have a load of DNA data, you could annotate the tree by having 
the dict associate PhyNodes with Ts - in this case DNA sequence types.
Then if you had some other data like comments, you'd make another such 
dictionary, where T would be String, and so on.

Best,
Ben.

On Tuesday, February 24, 2015 at 1:30:57 AM UTC, Ben Ward wrote:

 Hi,

 In Julia, if you are not sure of an arguments type beforehand, you can use 
 'Any', or if you know it comes under an abstract type, you can use that.

 I've been working on a type for BioJulia which allows for representation 
 of a node of a phylogenetic tree that will allow annotation with biological 
 data in the form of other types from the packages from BioJulia:

 @doc 
 PhyExtension allows defining arbitrary metadata to annotate nodes.


 This allows the PhyNode type to support any phylogenetic tree format 
 that includes annotations (e.g. PhyloXML, NeXML), and allows programmatic 
 extension of nodes with annotations.
  -
 type PhyExtension{T}
   value::T
 end


 @doc 
 PhyNode represents a node in a phylogenetic tree.


 A node can have:


 - `name`
 - `branchlength`
 - one or more `extensions`
 - a reference to its `parent` PhyNode
 - reference to one or more `children`
  -
 type PhyNode
   name::String
   branchlength::Float64
   confidence::Float64
   extensions::Vector{PhyExtension}
   children::Vector{PhyNode}
   parent::PhyNode
  # Inner constructor goes here.

 As you can see - PhyExtension is parametric so can be constructed to 
 contain any type.

 I'm wondering, though is this the best that can be done, can I handle the 
 potential to hold any number of types with unknown variables any better - 
 if so, how?

 Is specifying a Vector of PhyExtension, where a PhyExtenision is a type 
 that contains any other type really giving that much information to the 
 Julia system to work on efficiently? 

 My understanding is that PhyExtension is parametric so a version is 
 effectively defined for every possible {T} (including user defined 
 composite types they might come up with). But then I imagine that's not 
 much information when creating the Vector{PhyExtension} as an element could 
 be any of the many possible PhyExtensions - the compiler has to allow for 
 an array that can store any PhyExtension? Or is it that the array contains 
 references to PhyExtensions so actually the array is simple?

 Thanks,
 Ben.

Re: [julia-users] Making optional fields of unknown type more efficient

[Ben Ward]

I see, I've hardcoded the most common attributes stored in nodes into the
nodes themselves - branch-lengths and confidence values and so on. I can
see this get's mind-bending in cases where nodes have different types and
so on. I wonder if a different approach is better - I've just had an idea
to do this differently, which might be a lot better. I'll write up some
solid code later and post back.



On Tue, Feb 24, 2015 at 8:25 AM, Mauro mauro...@runbox.com wrote:

 Sorry my first sentence should have read:
  Julia will *not* be able to produce specialised code for any
 non-concrete type
  in your PhyNode.


  It does not matter whether it's Any or some other
  abstract/parameter-less type.  At least that is how I read the docs:
 
 http://docs.julialang.org/en/latest/manual/performance-tips/#type-declarations
 
  However, I think a method will only be slow if it actually uses the
  PhyNode.extensions field, otherwise it should be fine.
 
  Anyway, can you not construct your type like so:
  type PhyExtension{T}
value::T
  end
 
  type PhyNode{T}
name::String
branchlength::Float64
confidence::Float64
extensions::Vector{PhyExtension{T}}
children::Vector{PhyNode{T}}
parent::PhyNode{T}
  end
 
  If a user needs to have different types of things in PhyNode.extensions
  she can choose PhyExtension{Any} and loose some performance, otherwise a
  concrete type, like PhyExtension{Float64}, will be fast.
 
  On Tue, 2015-02-24 at 02:30, Ben Ward axolotlfan9...@gmail.com wrote:
  Hi,
 
  In Julia, if you are not sure of an arguments type beforehand, you can
 use
  'Any', or if you know it comes under an abstract type, you can use that.
 
  I've been working on a type for BioJulia which allows for
 representation of
  a node of a phylogenetic tree that will allow annotation with biological
  data in the form of other types from the packages from BioJulia:
 
  @doc 
  PhyExtension allows defining arbitrary metadata to annotate nodes.
 
 
  this allows the PhyNode type to support any phylogenetic tree format
  that includes annotations (e.g. PhyloXML, NeXML), and allows
 programmatic
  extension of nodes with annotations.
   -
  type PhyExtension{T}
value::T
  end
 
 
  @doc 
  PhyNode represents a node in a phylogenetic tree.
 
 
  A node can have:
 
 
  - `name`
  - `branchlength`
  - one or more `extensions`
  - a reference to its `parent` PhyNode
  - reference to one or more `children`
   -
  type PhyNode
name::String
branchlength::Float64
confidence::Float64
extensions::Vector{PhyExtension}
children::Vector{PhyNode}
parent::PhyNode
   # Inner constructor goes here.
 
  As you can see - PhyExtension is parametric so can be constructed to
  contain any type.
 
  I'm wondering, though is this the best that can be done, can I handle
 the
  potential to hold any number of types with unknown variables any better
 -
  if so, how?
 
  Is specifying a Vector of PhyExtension, where a PhyExtenision is a type
  that contains any other type really giving that much information to the
  Julia system to work on efficiently?
 
  My understanding is that PhyExtension is parametric so a version is
  effectively defined for every possible {T} (including user defined
  composite types they might come up with). But then I imagine that's not
  much information when creating the Vector{PhyExtension} as an element
 could
  be any of the many possible PhyExtensions - the compiler has to allow
 for
  an array that can store any PhyExtension? Or is it that the array
 contains
  references to PhyExtensions so actually the array is simple?
 
  Thanks,
  Ben.

[julia-users] Making optional fields of unknown type more efficient

[Ben Ward]

Hi,

In Julia, if you are not sure of an arguments type beforehand, you can use 
'Any', or if you know it comes under an abstract type, you can use that.

I've been working on a type for BioJulia which allows for representation of 
a node of a phylogenetic tree that will allow annotation with biological 
data in the form of other types from the packages from BioJulia:

@doc 
PhyExtension allows defining arbitrary metadata to annotate nodes.


This allows the PhyNode type to support any phylogenetic tree format 
that includes annotations (e.g. PhyloXML, NeXML), and allows programmatic 
extension of nodes with annotations.
 -
type PhyExtension{T}
  value::T
end


@doc 
PhyNode represents a node in a phylogenetic tree.


A node can have:


- `name`
- `branchlength`
- one or more `extensions`
- a reference to its `parent` PhyNode
- reference to one or more `children`
 -
type PhyNode
  name::String
  branchlength::Float64
  confidence::Float64
  extensions::Vector{PhyExtension}
  children::Vector{PhyNode}
  parent::PhyNode
 # Inner constructor goes here.

As you can see - PhyExtension is parametric so can be constructed to 
contain any type.

I'm wondering, though is this the best that can be done, can I handle the 
potential to hold any number of types with unknown variables any better - 
if so, how?

Is specifying a Vector of PhyExtension, where a PhyExtenision is a type 
that contains any other type really giving that much information to the 
Julia system to work on efficiently? 

My understanding is that PhyExtension is parametric so a version is 
effectively defined for every possible {T} (including user defined 
composite types they might come up with). But then I imagine that's not 
much information when creating the Vector{PhyExtension} as an element could 
be any of the many possible PhyExtensions - the compiler has to allow for 
an array that can store any PhyExtension? Or is it that the array contains 
references to PhyExtensions so actually the array is simple?

Thanks,
Ben.

[julia-users] Finding common values in arrays.

[Ben Ward]

Given a variable number of arrays, or an array of arrays, what is the best 
Julian way to find the values that occur in all of them? So for

a = [1,2,3,4,5]
b = [6,7,3,4,9]

would return 3 and 4?

I thought about using .== for every possible pair of the N arrays but I 
wondered if there was a better way?

Thanks,
Ben.

[julia-users] Creation of custom iterators.

[Ben Ward]

Hi,

I've been writing a type for recursive tree structures, and several types 
that traverse that tree in various manners like breadth first or depth 
first. They have their own methods for getting the current tree node, 
moving to the next node, whether an end has been reached and so on. The 
contain fields for the nodes several steps ahead, those past etc. I 
wondered if I might make it so as these types might easier be used in loops 
by giving them the iterator protocol methods? I've not seen how to define 
custom operators, is it as simple as defining start next and done? How is 
the current value gotten? I guess its returned by next(). 

Thanks,
Ben.  

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

I'm not nessecerily trying it iterate over the children of a node. Rather I 
have defined a series of types that facilitate traversing a tree in various 
ways for my Phylogenetics.jl package, for example by depth first:

type TraverserCore
  Start::PhyNode
  Behind::Stack
  History::Array{PhyNode, 1}
  Current::PhyNode
end


type DepthFirstTraverser : TreeTraverser
  Ahead::Stack
  Core::TraverserCore
  function DepthFirstTraverser(tree::Phylogeny)
x = new(Stack(PhyNode), TraverserCore(tree.Root, Stack(PhyNode), PhyNode
[], tree.Root))
for i in x.Core.Current.Children
  push!(x.Ahead, i)
end
return x
  end
end


It has methods like:


function next!(x::DepthFirstTraverser)
  push!(x.Core.Behind, x.Core.Current)
  x.Core.Current = pop!(x.Ahead)
  for i in x.Core.Current.Children
push!(x.Ahead, i)
  end
end


function getCurrent(x::TreeTraverser)
  return x.Core.Current
end


function hasReachedEnd(x::TreeTraverser)
  length(x.Ahead)  0 ? false : true
end


Which seem similar to start, next, and done. I'd use them in a loop like so 
again from Phylogenetics.jl:

while true
show(getCurrent(traverser))
if hasReachedEnd(traverser)
  break
end
next!(traverser)
end

But I'd like to make it behave more like an iterator - so be able to define 
the iterator methods for it so I can do something like

for i = DepthFirstTraverser(myTree)
# BLARGH
end

And it will be translated accordingly. I think this is doable by defining 
the three methods, making use of the types the method already has.

The idea is to have a load of types that allow the user to code iteration 
over the tree in any possible way, easily, providing there is a 
TreeTraverser type for it.

Best,
Ben.


On Sunday, July 27, 2014 2:14:38 PM UTC+1, Tim Holy wrote:

 for x in obj 
 # blah 
 end 

 will iterate if you've defined start, next, and done functions for which 
 the 
 first argument has typeof(obj). In your case you'd presumably use a node 
 as 
 obj, and the traversal would be recursively over all children of that 
 node. 

 If you want a specific tree example, check out ProfileView.jl/src/tree.jl. 

 Best, 
 --Tim 

 On Sunday, July 27, 2014 05:13:39 AM Ben Ward wrote: 
  Hi, 
  
  I've been writing a type for recursive tree structures, and several 
 types 
  that traverse that tree in various manners like breadth first or depth 
  first. They have their own methods for getting the current tree node, 
  moving to the next node, whether an end has been reached and so on. The 
  contain fields for the nodes several steps ahead, those past etc. I 
  wondered if I might make it so as these types might easier be used in 
 loops 
  by giving them the iterator protocol methods? I've not seen how to 
 define 
  custom operators, is it as simple as defining start next and done? How 
 is 
  the current value gotten? I guess its returned by next(). 
  
  Thanks, 
  Ben. 

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

My traverser types are not exactly wrappers quite a simple as they contain 
FIFO and FILO structures that keep track of things - I struggle to imagine 
how else to have them. Do the three iterate methods necessarily need to 
have the second argument state? My types know they are done - 
hasReachedEnd() - because there are no more nodes to visit in their Ahead 
Queue/Stack. So would a done() that only requires the type be sufficient 
with no state input variable as in done(tier, state)?

Best,
Ben.

On Sunday, July 27, 2014 4:49:24 PM UTC+1, Tim Holy wrote:

 You can obtain different types of iteration simply by wrapping obj in 
 different thin-wrappers. For example, you can define 

 immutable SomeOtherWayOfTraversing{T} 
 obj::T 
 end 

 which is used as 

 for x in SomeOtherWayOfTraversing(obj) 
 # blah 
 end 

 and then write the specific start, next, done methods like this: 

 start{T}(iter::SomeOtherWayOfTraversing{T}) 

 You can get totally different behavior this way from what would happen 
 when you 
 just say for x in obj 


 You might want to browse through more packages to see more examples. 
 Here's 
 one: 

 https://github.com/timholy/Grid.jl/blob/600cbcf645a73525fb6d563d5a148b9d8b2668aa/src/counter.jl
  
 but many other packages (DataFrames, Gtk, HDF5, etc) define iterators. 

 --Tim 

 On Sunday, July 27, 2014 06:41:43 AM Ben Ward wrote: 
  I'm not nessecerily trying it iterate over the children of a node. 
 Rather I 
  have defined a series of types that facilitate traversing a tree in 
 various 
  ways for my Phylogenetics.jl package, for example by depth first: 
  
  type TraverserCore 
Start::PhyNode 
Behind::Stack 
History::Array{PhyNode, 1} 
Current::PhyNode 
  end 
  
  
  type DepthFirstTraverser : TreeTraverser 
Ahead::Stack 
Core::TraverserCore 
function DepthFirstTraverser(tree::Phylogeny) 
  x = new(Stack(PhyNode), TraverserCore(tree.Root, Stack(PhyNode), 
 PhyNode 
  [], tree.Root)) 
  for i in x.Core.Current.Children 
push!(x.Ahead, i) 
  end 
  return x 
end 
  end 
  
  
  It has methods like: 
  
  
  function next!(x::DepthFirstTraverser) 
push!(x.Core.Behind, x.Core.Current) 
x.Core.Current = pop!(x.Ahead) 
for i in x.Core.Current.Children 
  push!(x.Ahead, i) 
end 
  end 
  
  
  function getCurrent(x::TreeTraverser) 
return x.Core.Current 
  end 
  
  
  function hasReachedEnd(x::TreeTraverser) 
length(x.Ahead)  0 ? false : true 
  end 
  
  
  Which seem similar to start, next, and done. I'd use them in a loop like 
 so 
  again from Phylogenetics.jl: 
  
  while true 
  show(getCurrent(traverser)) 
  if hasReachedEnd(traverser) 
break 
  end 
  next!(traverser) 
  end 
  
  But I'd like to make it behave more like an iterator - so be able to 
 define 
  the iterator methods for it so I can do something like 
  
  for i = DepthFirstTraverser(myTree) 
  # BLARGH 
  end 
  
  And it will be translated accordingly. I think this is doable by 
 defining 
  the three methods, making use of the types the method already has. 
  
  The idea is to have a load of types that allow the user to code 
 iteration 
  over the tree in any possible way, easily, providing there is a 
  TreeTraverser type for it. 
  
  Best, 
  Ben. 
  
  On Sunday, July 27, 2014 2:14:38 PM UTC+1, Tim Holy wrote: 
   for x in obj 
   
   # blah 
   
   end 
   
   will iterate if you've defined start, next, and done functions for 
 which 
   the 
   first argument has typeof(obj). In your case you'd presumably use a 
 node 
   as 
   obj, and the traversal would be recursively over all children of that 
   node. 
   
   If you want a specific tree example, check out 
 ProfileView.jl/src/tree.jl. 
   
   Best, 
   --Tim 
   
   On Sunday, July 27, 2014 05:13:39 AM Ben Ward wrote: 
Hi, 

I've been writing a type for recursive tree structures, and several 
   
   types 
   
that traverse that tree in various manners like breadth first or 
 depth 
first. They have their own methods for getting the current tree 
 node, 
moving to the next node, whether an end has been reached and so on. 
 The 
contain fields for the nodes several steps ahead, those past etc. I 
wondered if I might make it so as these types might easier be used 
 in 
   
   loops 
   
by giving them the iterator protocol methods? I've not seen how to 
   
   define 
   
custom operators, is it as simple as defining start next and done? 
 How 
   
   is 
   
the current value gotten? I guess its returned by next(). 

Thanks, 
Ben. 

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

I had not considered this - so state variable is a complex type which would 
have say the Queue/Stack and current value, and the start, next and done 
methods update it?

On Sunday, July 27, 2014 7:48:56 PM UTC+1, Tim Holy wrote:

 Why can't you keep track of everything in the state variable, and make 
 your 
 iterator-types trivial? 

 --Tim 

 On Sunday, July 27, 2014 11:07:36 AM Ben Ward wrote: 
  My traverser types are not exactly wrappers quite a simple as they 
 contain 
  FIFO and FILO structures that keep track of things - I struggle to 
 imagine 
  how else to have them. Do the three iterate methods necessarily need to 
  have the second argument state? My types know they are done - 
  hasReachedEnd() - because there are no more nodes to visit in their 
 Ahead 
  Queue/Stack. So would a done() that only requires the type be sufficient 
  with no state input variable as in done(tier, state)? 
  
  Best, 
  Ben. 
  
  On Sunday, July 27, 2014 4:49:24 PM UTC+1, Tim Holy wrote: 
   You can obtain different types of iteration simply by wrapping obj 
 in 
   different thin-wrappers. For example, you can define 
   
   immutable SomeOtherWayOfTraversing{T} 
   
   obj::T 
   
   end 
   
   which is used as 
   
   for x in SomeOtherWayOfTraversing(obj) 
   
   # blah 
   
   end 
   
   and then write the specific start, next, done methods like this: 
   
   start{T}(iter::SomeOtherWayOfTraversing{T}) 
   
   You can get totally different behavior this way from what would happen 
   when you 
   just say for x in obj 
   
   
   You might want to browse through more packages to see more examples. 
   Here's 
   one: 
   
   
 https://github.com/timholy/Grid.jl/blob/600cbcf645a73525fb6d563d5a148b9d8b 
   2668aa/src/counter.jl but many other packages (DataFrames, Gtk, HDF5, 
 etc) 
   define iterators. 
   
   --Tim 
   
   On Sunday, July 27, 2014 06:41:43 AM Ben Ward wrote: 
I'm not nessecerily trying it iterate over the children of a node. 
   
   Rather I 
   
have defined a series of types that facilitate traversing a tree in 
   
   various 
   
ways for my Phylogenetics.jl package, for example by depth first: 

type TraverserCore 

  Start::PhyNode 
  Behind::Stack 
  History::Array{PhyNode, 1} 
  Current::PhyNode 

end 


type DepthFirstTraverser : TreeTraverser 

  Ahead::Stack 
  Core::TraverserCore 
  function DepthFirstTraverser(tree::Phylogeny) 
  
x = new(Stack(PhyNode), TraverserCore(tree.Root, Stack(PhyNode), 
   
   PhyNode 
   
[], tree.Root)) 

for i in x.Core.Current.Children 

  push!(x.Ahead, i) 

end 
return x 
  
  end 

end 


It has methods like: 


function next!(x::DepthFirstTraverser) 

  push!(x.Core.Behind, x.Core.Current) 
  x.Core.Current = pop!(x.Ahead) 
  for i in x.Core.Current.Children 
  
push!(x.Ahead, i) 
  
  end 

end 


function getCurrent(x::TreeTraverser) 

  return x.Core.Current 

end 


function hasReachedEnd(x::TreeTraverser) 

  length(x.Ahead)  0 ? false : true 

end 


Which seem similar to start, next, and done. I'd use them in a loop 
 like 
   
   so 
   
again from Phylogenetics.jl: 

while true 

show(getCurrent(traverser)) 
if hasReachedEnd(traverser) 

  break 

end 
next!(traverser) 

end 

But I'd like to make it behave more like an iterator - so be able to 
   
   define 
   
the iterator methods for it so I can do something like 

for i = DepthFirstTraverser(myTree) 
# BLARGH 
end 

And it will be translated accordingly. I think this is doable by 
   
   defining 
   
the three methods, making use of the types the method already has. 

The idea is to have a load of types that allow the user to code 
   
   iteration 
   
over the tree in any possible way, easily, providing there is a 
TreeTraverser type for it. 

Best, 
Ben. 

On Sunday, July 27, 2014 2:14:38 PM UTC+1, Tim Holy wrote: 
 for x in obj 
 
 # blah 
 
 end 
 
 will iterate if you've defined start, next, and done functions for 
   
   which 
   
 the 
 first argument has typeof(obj). In your case you'd presumably use 
 a 
   
   node 
   
 as 
 obj, and the traversal would be recursively over all children of 
 that 
 node. 
 
 If you want a specific tree example, check out 
   
   ProfileView.jl/src/tree.jl. 
   
 Best, 
 --Tim 
 
 On Sunday, July 27, 2014 05:13:39 AM Ben Ward wrote: 
  Hi, 
  
  I've been writing a type for recursive tree structures, and 
 several 
 
 types 
 
  that traverse that tree in various

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

Yesh I'm reading through Iterators.jl now to try and get my head around it.

On Sunday, July 27, 2014 7:58:09 PM UTC+1, Tim Holy wrote:

 Did you check out the examples I suggested? :) 

 On Sunday, July 27, 2014 11:56:16 AM Ben Ward wrote: 
  I had not considered this - so state variable is a complex type which 
 would 
  have say the Queue/Stack and current value, and the start, next and done 
  methods update it? 
  
  On Sunday, July 27, 2014 7:48:56 PM UTC+1, Tim Holy wrote: 
   Why can't you keep track of everything in the state variable, and make 
   your 
   iterator-types trivial? 
   
   --Tim 
   
   On Sunday, July 27, 2014 11:07:36 AM Ben Ward wrote: 
My traverser types are not exactly wrappers quite a simple as they 
   
   contain 
   
FIFO and FILO structures that keep track of things - I struggle to 
   
   imagine 
   
how else to have them. Do the three iterate methods necessarily need 
 to 
have the second argument state? My types know they are done - 
hasReachedEnd() - because there are no more nodes to visit in their 
   
   Ahead 
   
Queue/Stack. So would a done() that only requires the type be 
 sufficient 
with no state input variable as in done(tier, state)? 

Best, 
Ben. 

On Sunday, July 27, 2014 4:49:24 PM UTC+1, Tim Holy wrote: 
 You can obtain different types of iteration simply by wrapping 
 obj 
   
   in 
   
 different thin-wrappers. For example, you can define 
 
 immutable SomeOtherWayOfTraversing{T} 
 
 obj::T 
 
 end 
 
 which is used as 
 
 for x in SomeOtherWayOfTraversing(obj) 
 
 # blah 
 
 end 
 
 and then write the specific start, next, done methods like this: 
 
 start{T}(iter::SomeOtherWayOfTraversing{T}) 
 
 You can get totally different behavior this way from what would 
 happen 
 when you 
 just say for x in obj 
 
 
 You might want to browse through more packages to see more 
 examples. 
 Here's 
   
 one: 
   
 https://github.com/timholy/Grid.jl/blob/600cbcf645a73525fb6d563d5a148b9d8b 
   
 2668aa/src/counter.jl but many other packages (DataFrames, Gtk, 
 HDF5, 
   
   etc) 
   
 define iterators. 
 
 --Tim 
 
 On Sunday, July 27, 2014 06:41:43 AM Ben Ward wrote: 
  I'm not nessecerily trying it iterate over the children of a 
 node. 
 
 Rather I 
 
  have defined a series of types that facilitate traversing a tree 
 in 
 
 various 
 
  ways for my Phylogenetics.jl package, for example by depth 
 first: 
  
  type TraverserCore 
  
Start::PhyNode 
Behind::Stack 
History::Array{PhyNode, 1} 
Current::PhyNode 
  
  end 
  
  
  type DepthFirstTraverser : TreeTraverser 
  
Ahead::Stack 
Core::TraverserCore 
function DepthFirstTraverser(tree::Phylogeny) 

  x = new(Stack(PhyNode), TraverserCore(tree.Root, 
 Stack(PhyNode), 
 
 PhyNode 
 
  [], tree.Root)) 
  
  for i in x.Core.Current.Children 
  
push!(x.Ahead, i) 
  
  end 
  return x 

end 
  
  end 
  
  
  It has methods like: 
  
  
  function next!(x::DepthFirstTraverser) 
  
push!(x.Core.Behind, x.Core.Current) 
x.Core.Current = pop!(x.Ahead) 
for i in x.Core.Current.Children 

  push!(x.Ahead, i) 

end 
  
  end 
  
  
  function getCurrent(x::TreeTraverser) 
  
return x.Core.Current 
  
  end 
  
  
  function hasReachedEnd(x::TreeTraverser) 
  
length(x.Ahead)  0 ? false : true 
  
  end 
  
  
  Which seem similar to start, next, and done. I'd use them in a 
 loop 
   
   like 
   
 so 
 
  again from Phylogenetics.jl: 
  
  while true 
  
  show(getCurrent(traverser)) 
  if hasReachedEnd(traverser) 
  
break 
  
  end 
  next!(traverser) 
  
  end 
  
  But I'd like to make it behave more like an iterator - so be 
 able to 
 
 define 
 
  the iterator methods for it so I can do something like 
  
  for i = DepthFirstTraverser(myTree) 
  # BLARGH 
  end 
  
  And it will be translated accordingly. I think this is doable by 
 
 defining 
 
  the three methods, making use of the types the method already 
 has. 
  
  The idea is to have a load of types that allow the user to code 
 
 iteration 
 
  over the tree in any possible way, easily, providing there is a 
  TreeTraverser type for it. 
  
  Best, 
  Ben. 
  
  On Sunday, July 27, 2014 2:14:38 PM UTC+1, Tim Holy wrote: 
   for x in obj

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

I've given this a go but it does not quite work as expected:

immutable DepthFirst
tree::Phylogeny
end

function start(x::DepthFirst)
  state = Stack(PhyNode)
  push!(state, x.tree.Root)
  return state
end

function next(x::DepthFirst, state)
  current::PhyNode = pop!(state)
  for i in current.Children
push!(state, i)
  end
  return current, state
end

function done(x::DepthFirst, state)
  return length(state) == 0 ? true : false
end

Then:

*for i in DepthFirst(myTree)*

*i*

*end*

results in:

*ERROR: `start` has no method matching start(::DepthFirst)*




* in anonymous at no file*

I'm not sure why this is - I have a method defined start() for the 
utterable immutable DepthFirst trivial type. I'm clearly missing something 
here.

On Sunday, July 27, 2014 7:58:09 PM UTC+1, Tim Holy wrote:

 Did you check out the examples I suggested? :) 

 On Sunday, July 27, 2014 11:56:16 AM Ben Ward wrote: 
  I had not considered this - so state variable is a complex type which 
 would 
  have say the Queue/Stack and current value, and the start, next and done 
  methods update it? 
  
  On Sunday, July 27, 2014 7:48:56 PM UTC+1, Tim Holy wrote: 
   Why can't you keep track of everything in the state variable, and make 
   your 
   iterator-types trivial? 
   
   --Tim 
   
   On Sunday, July 27, 2014 11:07:36 AM Ben Ward wrote: 
My traverser types are not exactly wrappers quite a simple as they 
   
   contain 
   
FIFO and FILO structures that keep track of things - I struggle to 
   
   imagine 
   
how else to have them. Do the three iterate methods necessarily need 
 to 
have the second argument state? My types know they are done - 
hasReachedEnd() - because there are no more nodes to visit in their 
   
   Ahead 
   
Queue/Stack. So would a done() that only requires the type be 
 sufficient 
with no state input variable as in done(tier, state)? 

Best, 
Ben. 

On Sunday, July 27, 2014 4:49:24 PM UTC+1, Tim Holy wrote: 
 You can obtain different types of iteration simply by wrapping 
 obj 
   
   in 
   
 different thin-wrappers. For example, you can define 
 
 immutable SomeOtherWayOfTraversing{T} 
 
 obj::T 
 
 end 
 
 which is used as 
 
 for x in SomeOtherWayOfTraversing(obj) 
 
 # blah 
 
 end 
 
 and then write the specific start, next, done methods like this: 
 
 start{T}(iter::SomeOtherWayOfTraversing{T}) 
 
 You can get totally different behavior this way from what would 
 happen 
 when you 
 just say for x in obj 
 
 
 You might want to browse through more packages to see more 
 examples. 
 Here's 
   
 one: 
   
 https://github.com/timholy/Grid.jl/blob/600cbcf645a73525fb6d563d5a148b9d8b 
   
 2668aa/src/counter.jl but many other packages (DataFrames, Gtk, 
 HDF5, 
   
   etc) 
   
 define iterators. 
 
 --Tim 
 
 On Sunday, July 27, 2014 06:41:43 AM Ben Ward wrote: 
  I'm not nessecerily trying it iterate over the children of a 
 node. 
 
 Rather I 
 
  have defined a series of types that facilitate traversing a tree 
 in 
 
 various 
 
  ways for my Phylogenetics.jl package, for example by depth 
 first: 
  
  type TraverserCore 
  
Start::PhyNode 
Behind::Stack 
History::Array{PhyNode, 1} 
Current::PhyNode 
  
  end 
  
  
  type DepthFirstTraverser : TreeTraverser 
  
Ahead::Stack 
Core::TraverserCore 
function DepthFirstTraverser(tree::Phylogeny) 

  x = new(Stack(PhyNode), TraverserCore(tree.Root, 
 Stack(PhyNode), 
 
 PhyNode 
 
  [], tree.Root)) 
  
  for i in x.Core.Current.Children 
  
push!(x.Ahead, i) 
  
  end 
  return x 

end 
  
  end 
  
  
  It has methods like: 
  
  
  function next!(x::DepthFirstTraverser) 
  
push!(x.Core.Behind, x.Core.Current) 
x.Core.Current = pop!(x.Ahead) 
for i in x.Core.Current.Children 

  push!(x.Ahead, i) 

end 
  
  end 
  
  
  function getCurrent(x::TreeTraverser) 
  
return x.Core.Current 
  
  end 
  
  
  function hasReachedEnd(x::TreeTraverser) 
  
length(x.Ahead)  0 ? false : true 
  
  end 
  
  
  Which seem similar to start, next, and done. I'd use them in a 
 loop 
   
   like 
   
 so 
 
  again from Phylogenetics.jl: 
  
  while true 
  
  show(getCurrent(traverser)) 
  if hasReachedEnd(traverser) 
  
break 
  
  end 
  next!(traverser) 
  
  end 
  
  But I'd like to make it behave more like an iterator - so be 
 able

Re: [julia-users] Creation of custom iterators.

[Ben Ward]

I see, if I put this into a module in a package will this still be 
nessecery? I read that when defining a module, base is implicitly imported.

On Sunday, July 27, 2014 8:58:33 PM UTC+1, Leah Hanson wrote:

 You need to either `import Base.start` or implement a function named 
 `Base.start` instead of `start`. That change will make your `start` 
 function extend the one from Base with new methods, rather than being a 
 separate function that happens to be named `start`. (This is a super common 
 confusion.)

 -- Leah


 On Sun, Jul 27, 2014 at 2:40 PM, Ben Ward axolotl...@gmail.com 
 javascript: wrote:

 I've given this a go but it does not quite work as expected:

 immutable DepthFirst
 tree::Phylogeny
 end

 function start(x::DepthFirst)
   state = Stack(PhyNode)
   push!(state, x.tree.Root)
   return state
 end

 function next(x::DepthFirst, state)
   current::PhyNode = pop!(state)
   for i in current.Children
 push!(state, i)
   end
   return current, state
 end

 function done(x::DepthFirst, state)
   return length(state) == 0 ? true : false
 end

 Then:

 *for i in DepthFirst(myTree)*

 *i*

 *end*

 results in:

 *ERROR: `start` has no method matching start(::DepthFirst)*




 * in anonymous at no file*

 I'm not sure why this is - I have a method defined start() for the 
 utterable immutable DepthFirst trivial type. I'm clearly missing something 
 here.


 On Sunday, July 27, 2014 7:58:09 PM UTC+1, Tim Holy wrote:

 Did you check out the examples I suggested? :) 

 On Sunday, July 27, 2014 11:56:16 AM Ben Ward wrote: 
  I had not considered this - so state variable is a complex type which 
 would 
  have say the Queue/Stack and current value, and the start, next and 
 done 
  methods update it? 
  
  On Sunday, July 27, 2014 7:48:56 PM UTC+1, Tim Holy wrote: 
   Why can't you keep track of everything in the state variable, and 
 make 
   your 
   iterator-types trivial? 
   
   --Tim 
   
   On Sunday, July 27, 2014 11:07:36 AM Ben Ward wrote: 
My traverser types are not exactly wrappers quite a simple as they 
   
   contain 
   
FIFO and FILO structures that keep track of things - I struggle to 
   
   imagine 
   
how else to have them. Do the three iterate methods necessarily 
 need to 
have the second argument state? My types know they are done - 
hasReachedEnd() - because there are no more nodes to visit in 
 their 
   
   Ahead 
   
Queue/Stack. So would a done() that only requires the type be 
 sufficient 
with no state input variable as in done(tier, state)? 

Best, 
Ben. 

On Sunday, July 27, 2014 4:49:24 PM UTC+1, Tim Holy wrote: 
 You can obtain different types of iteration simply by wrapping 
 obj 
   
   in 
   
 different thin-wrappers. For example, you can define 
 
 immutable SomeOtherWayOfTraversing{T} 
 
 obj::T 
 
 end 
 
 which is used as 
 
 for x in SomeOtherWayOfTraversing(obj) 
 
 # blah 
 
 end 
 
 and then write the specific start, next, done methods like this: 
 
 start{T}(iter::SomeOtherWayOfTraversing{T}) 
 
 You can get totally different behavior this way from what would 
 happen 
 when you 
 just say for x in obj 
 
 
 You might want to browse through more packages to see more 
 examples. 
 Here's 
   
 one: 
   https://github.com/timholy/Grid.jl/blob/
 600cbcf645a73525fb6d563d5a148b9d8b 
   
 2668aa/src/counter.jl but many other packages (DataFrames, Gtk, 
 HDF5, 
   
   etc) 
   
 define iterators. 
 
 --Tim 
 
 On Sunday, July 27, 2014 06:41:43 AM Ben Ward wrote: 
  I'm not nessecerily trying it iterate over the children of a 
 node. 
 
 Rather I 
 
  have defined a series of types that facilitate traversing a 
 tree in 
 
 various 
 
  ways for my Phylogenetics.jl package, for example by depth 
 first: 
  
  type TraverserCore 
  
Start::PhyNode 
Behind::Stack 
History::Array{PhyNode, 1} 
Current::PhyNode 
  
  end 
  
  
  type DepthFirstTraverser : TreeTraverser 
  
Ahead::Stack 
Core::TraverserCore 
function DepthFirstTraverser(tree::Phylogeny) 

  x = new(Stack(PhyNode), TraverserCore(tree.Root, 
 Stack(PhyNode), 
 
 PhyNode 
 
  [], tree.Root)) 
  
  for i in x.Core.Current.Children 
  
push!(x.Ahead, i) 
  
  end 
  return x 

end 
  
  end 
  
  
  It has methods like: 
  
  
  function next!(x::DepthFirstTraverser) 
  
push!(x.Core.Behind, x.Core.Current) 
x.Core.Current = pop!(x.Ahead) 
for i in x.Core.Current.Children 

  push!(x.Ahead, i) 

end 
  
  end 
  
  
  function getCurrent(x::TreeTraverser

[julia-users] Checking for Undefined Reference.

[Ben Ward]

I have a type that contains references to other instances of the same type 
- like a doubly linked list or - as it's intended use, like a tree like 
structure. They contain references to a parent of the same types and an 
array of references to children of the same type. However a root or singly 
constructed instance of the type will not start out with a parent or 
children and so these fields will be #undef, how can I check for #undef 
fields? It didn't say in the types section in the manual, although it did 
say accessing it is an automatic error so I guess a try block might work - 
but I don't know if try blocks are preferable in Julia to simply 
anticipating the #undef. I've normally heard the mantra of anticipating an 
expected exception rather than make excessive try catches.

Thanks. 

[julia-users] String from a substring

[Ben Ward]

This is probably a simple question, I've done a string splitting operation 
with split(), which has given me an array of SubString types, but I want 
them to be of type String so as I can feed the bits of the split up string 
to some method that requires a String type. string(*substring type*) does 
not work.

Thanks,
Ben.

Re: [julia-users] String from a substring

[Ben Ward]

I'm calling the constructor go a type I made, there's a string field is 
accepts to fill a string field in the type.

On Monday, July 21, 2014 5:17:25 PM UTC+1, Leah Hanson wrote:

 SubString is a subtype of String, so any function accepting a String 
 should accept a SubString. What function are you calling where this doesn't 
 work?

 -- Leah


 On Mon, Jul 21, 2014 at 11:13 AM, Ben Ward axolotl...@gmail.com 
 javascript: wrote:

 This is probably a simple question, I've done a string splitting 
 operation with split(), which has given me an array of SubString types, but 
 I want them to be of type String so as I can feed the bits of the split up 
 string to some method that requires a String type. string(*substring type*) 
 does not work.

 Thanks,
 Ben.

Re: [julia-users] String from a substring

[Ben Ward]

I've just tested the constructor with a dummy example, and it works so 
obvious the error is elsewhere, I'll find it and update.


On Monday, July 21, 2014 5:17:25 PM UTC+1, Leah Hanson wrote:

 SubString is a subtype of String, so any function accepting a String 
 should accept a SubString. What function are you calling where this doesn't 
 work?

 -- Leah


 On Mon, Jul 21, 2014 at 11:13 AM, Ben Ward axolotl...@gmail.com 
 javascript: wrote:

 This is probably a simple question, I've done a string splitting 
 operation with split(), which has given me an array of SubString types, but 
 I want them to be of type String so as I can feed the bits of the split up 
 string to some method that requires a String type. string(*substring type*) 
 does not work.

 Thanks,
 Ben.

[julia-users] Specific methods for parametric types

[Ben Ward]

If I define a parametric type:

type MyType{T}
  firstval::T
  secondval::T
end

Can specific methods then be written for specific forms of MyType? For 
example a method that only worked for MyType containing Int64 as T? I 
haven’t tried this yet - I've run into my first real world application of 
parametric types.

Best,
Ben.

[julia-users] Re: Tagging a package

[Ben Ward]

Ok I found the issue relating to this problem:

https://github.com/JuliaLang/julia/issues/5998

I deleted the Julia Package manager token on GitHub as suggested and it now 
works!

On Thursday, July 3, 2014 3:57:20 AM UTC+1, Ben Ward wrote:

 Hi,

 I'm trying to update a package for the first time with the new package 
 manager commands like tag and so on:

 *julia **Pkg.publish()*

 *INFO: Validating METADATA*

 *INFO: Pushing Phylogenetics permanent tags: v0.0.2*

 Permission denied (publickey).

 fatal: Could not read from remote repository.



 Please make sure you have the correct access rights

 and the repository exists.

 *ERROR: failed process: Process(`git 
 --work-tree=/Users/axolotlfan9250/.julia/v0.3/Phylogenetics 
 --git-dir=/Users/axolotlfan9250/.julia/v0.3/Phylogenetics/.git push -q 
 origin refs/tags/v0.0.2:refs/tags/v0.0.2`, ProcessExited(128)) [128]*

 * in wait at ./task.jl:279*

 * in wait at ./task.jl:189*

 * in wait at task.jl:48*

 * in sync_end at ./task.jl:306*

 * in publish at pkg/entry.jl:314*

 * in anonymous at pkg/dir.jl:28*

 * in cd at file.jl:20*

 * in cd at pkg/dir.jl:28*

 * in publish at pkg.jl:57*



 I'm not 100% sure why I'm getting this, shouldn't it just send a PR to 
 METADATA? I'm not 100% clear on how this works - do I need a fork of 
 METADATA still? The process that fails looks like it's trying to push 
 something to ward9250/Phylogenetics.jl? In which case I wonder if it's a 
 naming issue Phylogenetics in .julia/v0.3/ and Phylogenetics.jl for the 
 repo?

[julia-users] Tagging a package

[Ben Ward]

Hi,

I'm trying to update a package for the first time with the new package 
manager commands like tag and so on:

*julia **Pkg.publish()*

*INFO: Validating METADATA*

*INFO: Pushing Phylogenetics permanent tags: v0.0.2*

Permission denied (publickey).

fatal: Could not read from remote repository.



Please make sure you have the correct access rights

and the repository exists.

*ERROR: failed process: Process(`git 
--work-tree=/Users/axolotlfan9250/.julia/v0.3/Phylogenetics 
--git-dir=/Users/axolotlfan9250/.julia/v0.3/Phylogenetics/.git push -q 
origin refs/tags/v0.0.2:refs/tags/v0.0.2`, ProcessExited(128)) [128]*

* in wait at ./task.jl:279*

* in wait at ./task.jl:189*

* in wait at task.jl:48*

* in sync_end at ./task.jl:306*

* in publish at pkg/entry.jl:314*

* in anonymous at pkg/dir.jl:28*

* in cd at file.jl:20*

* in cd at pkg/dir.jl:28*

* in publish at pkg.jl:57*



I'm not 100% sure why I'm getting this, shouldn't it just send a PR to 
METADATA? I'm not 100% clear on how this works - do I need a fork of 
METADATA still? The process that fails looks like it's trying to push 
something to ward9250/Phylogenetics.jl? In which case I wonder if it's a 
naming issue Phylogenetics in .julia/v0.3/ and Phylogenetics.jl for the 
repo?

Re: [julia-users] Macros inside functions

[Ben Ward]

Thanks, Jameson,

I should say in this case it is obvious the function would be better and I 
agree, this was a case I made myself to see if it could be done. The reason 
being, I'm writing a type, the composition of which could be extended by 
the user to contain other information (I'm working with a data format and 
file format(s) for phylogenetic data which is often non standard). So I 
made the base type extendible, and now I'm trying to get the stuff that 
reads or writes to file to be extendible. I'm currently working on the 
writing stuff to file, and what I've done got is a function which will 
write the data to file, including the extra user extended data, as long as 
the user has provided a method for representing it as a string, and so I 
thought providing some macro's to make defining such methods more quick and 
painless to write for the user, but perhaps providing a set of 
sub-functions would be better. Hence the INN macro, so you don't have to 
write lines checking it's not nothing every time (which I know 
shouldn't happen, but again, this is a bit of a contrived case to try 
and see if it could/should be done this way. I also had an idea for writing 
a macro which generated the entire function for the user given some basic 
information, but baby steps first. Strikes me a lot of times I figure a 
macro would be good for something, a function can do the same job.

Best,
Ben.



 

On Friday, April 18, 2014 6:38:14 AM UTC+1, Jameson wrote:

 disclaimer: I would use a function, not a macro for this purpose: 
 INN( outstring, value ) = value !== nothing ? $outstring$value : 
 outstring 
 INN(outstring, x.value) 

 it is simpler, and gives up nothing in exchange for actually being 
 readable and versatile 

 but assuming you had a more interesting use case: when you expand the 
 macro as written, the result is: 
 $string$value, with string=:outstring and value=:(x.value), exactly 
 as you specified: outstringx.value 

 but you want the string before interpolation (note that I'm switching 
 to the functional form of string interpolation because it is easier to 
 read in macro form), so we need this complicated thing to make a macro 
 that is actually just a function: 
 macro INN(str, val) 
   :( v = $(esc(val)); s = $(esc(str)); v != nothing ? string(v, s) : s ) 
 end 

 but the better way to write that is: 
 INN( outstring, value ) = value !== nothing ? $outstring$value : 
 outstring 
 macro INN(str, val) 
   :( INN($(esc(str)), $(esc(val))) ) 
 end 

 why is this better? it's easier to read, write, and debug (also, 
 because it makes it obvious that you should not be using a macro for 
 this) 

 finally, there's the even cleaner options: 
   outstring = MeltTemp$(x.value!==nothing?x.value:)/MeltTemp 

 or, using multiple dispatch -- my favorite option (excellent API for 
 future extensibility): 
   mystr(::Nothing) =  
   mystr(x) = x 
   outstring = MeltTemp$(mystr(x))/MeltTemp 

 note that since in your type, you have value::Int, so value will 
 always be an Int, never another type (like nothing), so it's a 
 pointless test 

 final note, string concatenation using $ or * or string (all of which 
 are exactly equivalent) is going to be slow compared to making an 
 IOBuffer, printing / writing / or showing all of your text to the 
 buffer, then calling takebuf_string(iobuf) at then end 

 On Thu, Apr 17, 2014 at 11:38 PM, Ben Ward 
 axolotl...@gmail.comjavascript: 
 wrote: 
  Hi Is it possible to use macros inside o functions in Julia if they are 
  expanded at compile time as the manual says? I've been playing with a 
 toy 
  type: 
  
  type MeltingTemperature 
value::Int 
  end 
  
  
  
  
  myvar = MeltingTemperature(56) 
  
  
  and say I want to make a function that write it out as an XML string: 
  MeltingTemperature56/MeltingTemperature 
  
  Now I want to make a function that constructs that string by 1). Adding 
 the 
  open tag 2). Checking the temperature is not nothing and if it is not, 
  including it in the string. 3). add the closing tag. 
  
  I figured step 2 would be a good candidate for a macro for a larger type 
 as 
  checking each value is not nothing would be tedious. So: 
  
  macro INN(string, value) 
value != nothing ? $string$value : $string 
  end 
  
  
  function writePhyXML(x::MeltingTemperature) 
outstring = MeltTemp 
outstring = @INN outstring x.value; 
outstring = $(outstring)/MeltTemp 
outstring 
  end 
  
  
  I figured the macro would be expanded and the function definition would 
  become: 
  
  function writePhyXML(x::MeltingTemperature) 
outstring = MeltTemp 
outstring = x.value != nothing ? $outstring$x.value : $outstring 
outstring = $(outstring)/MeltTemp 
outstring 
  end 
  
  
  But results in outstringx.value/MeltTemp 
  
  Is it possible to do this with macro's? I had presumed it is. 
  
  (I know it's possible by just using a sub function, but I wanted to see 
 how 
  to try and use macros

Re: [julia-users] Macros inside functions

[Ben Ward]

Thinking about it some more, and how you advise to use multiple dispatch to 
write these things out - is it possible to use multiple dispatch to read in 
things similarly? Because the thing you are reading in is to become the 
types you base the dispatch of the method on, so how could this apply to 
reading in types when the instances of the types are not extant yet? Maybe 
if you supply a empty instance of the type you want to read in for the 
dispatch? e.g.

inmystr(x::file, y::MeltingPoint) = # how to extract info from text here.

So I want to reading in MeltingPoint data from a file, and want to use the 
inmystr method that applies based on the function signature, yet I have no 
MeltingPoint types currently, because I'm trying to read them in, instead 
of having the object and writing it out. I know in C++ when trying to catch 
multiple different types of possible throws, you can define empty types to 
act as error types, I wonder if such a tactic here is applicable.

Best,
Ben.

On Friday, April 18, 2014 6:38:14 AM UTC+1, Jameson wrote:

 disclaimer: I would use a function, not a macro for this purpose: 
 INN( outstring, value ) = value !== nothing ? $outstring$value : 
 outstring 
 INN(outstring, x.value) 

 it is simpler, and gives up nothing in exchange for actually being 
 readable and versatile 

 but assuming you had a more interesting use case: when you expand the 
 macro as written, the result is: 
 $string$value, with string=:outstring and value=:(x.value), exactly 
 as you specified: outstringx.value 

 but you want the string before interpolation (note that I'm switching 
 to the functional form of string interpolation because it is easier to 
 read in macro form), so we need this complicated thing to make a macro 
 that is actually just a function: 
 macro INN(str, val) 
   :( v = $(esc(val)); s = $(esc(str)); v != nothing ? string(v, s) : s ) 
 end 

 but the better way to write that is: 
 INN( outstring, value ) = value !== nothing ? $outstring$value : 
 outstring 
 macro INN(str, val) 
   :( INN($(esc(str)), $(esc(val))) ) 
 end 

 why is this better? it's easier to read, write, and debug (also, 
 because it makes it obvious that you should not be using a macro for 
 this) 

 finally, there's the even cleaner options: 
   outstring = MeltTemp$(x.value!==nothing?x.value:)/MeltTemp 

 or, using multiple dispatch -- my favorite option (excellent API for 
 future extensibility): 
   mystr(::Nothing) =  
   mystr(x) = x 
   outstring = MeltTemp$(mystr(x))/MeltTemp 

 note that since in your type, you have value::Int, so value will 
 always be an Int, never another type (like nothing), so it's a 
 pointless test 

 final note, string concatenation using $ or * or string (all of which 
 are exactly equivalent) is going to be slow compared to making an 
 IOBuffer, printing / writing / or showing all of your text to the 
 buffer, then calling takebuf_string(iobuf) at then end 

 On Thu, Apr 17, 2014 at 11:38 PM, Ben Ward 
 axolotl...@gmail.comjavascript: 
 wrote: 
  Hi Is it possible to use macros inside o functions in Julia if they are 
  expanded at compile time as the manual says? I've been playing with a 
 toy 
  type: 
  
  type MeltingTemperature 
value::Int 
  end 
  
  
  
  
  myvar = MeltingTemperature(56) 
  
  
  and say I want to make a function that write it out as an XML string: 
  MeltingTemperature56/MeltingTemperature 
  
  Now I want to make a function that constructs that string by 1). Adding 
 the 
  open tag 2). Checking the temperature is not nothing and if it is not, 
  including it in the string. 3). add the closing tag. 
  
  I figured step 2 would be a good candidate for a macro for a larger type 
 as 
  checking each value is not nothing would be tedious. So: 
  
  macro INN(string, value) 
value != nothing ? $string$value : $string 
  end 
  
  
  function writePhyXML(x::MeltingTemperature) 
outstring = MeltTemp 
outstring = @INN outstring x.value; 
outstring = $(outstring)/MeltTemp 
outstring 
  end 
  
  
  I figured the macro would be expanded and the function definition would 
  become: 
  
  function writePhyXML(x::MeltingTemperature) 
outstring = MeltTemp 
outstring = x.value != nothing ? $outstring$x.value : $outstring 
outstring = $(outstring)/MeltTemp 
outstring 
  end 
  
  
  But results in outstringx.value/MeltTemp 
  
  Is it possible to do this with macro's? I had presumed it is. 
  
  (I know it's possible by just using a sub function, but I wanted to see 
 how 
  to try and use macros to generate code for function definitions if 
  possible.) 
  
  Thanks, 
  Ben. 

[julia-users] expandmacro() on function definitions.

[Ben Ward]

Hi, Is it possible to use expandmacro() on a function/method definition 
which contains within it a code generating macro? If I put the function 
definition in a begin end block, and assign it to a variable called code 
and then do expandmacro(code) it just tells me *(generic function with 1 
method),* but I'd like to see the function body with the macro expanded.

Thanks,
Ben.

[julia-users] Recursive function works without macro but does not work with it.

[Ben Ward]

Hi, I have quite a complex couple of functions and a long winded problem, 
I'll be as clear as I can be. I have a function, which builds up composite 
types based on the contents of a data file (XML). I use a recursive 
function to do this: (Note it's not 100% finished but it does successfully 
crawl the XML tree and create the basic structure and composite types, by 
starting at the element, building the required type from the info at that 
element, and then calling the same function recursively on all the children:
 
# xmlclade is a type from LightXML, cladeArray is the variable of interest 
and is what contains the results of each recursive call to this function. 
currentClade is a composite type containing an integer and so is mutable, 
it is used in recursive calls to the functions.  


function recursiveBuild(xmlclade, cladeArray, currentClade, 
parentClade::Int)
  # Update the node tracker.
  currentClade.nodeIndex += 1
  current = currentClade.nodeIndex # Initialize a local variable called 
current, taken from the currentClade variable to keep as the variable to 
pass to furthur recursive calls as the parent index.
  # Get name of clade element.
  label = 
  isroot = currentClade == 0 ? true : false
  children = get_elements_by_tagname(xmlclade, clade)
  istip = length(children) == 0 ? true : false
  # Make the clade object by calling the constructor.
  cladeArray[currentClade.nodeIndex] = PhyXElement(label, isroot, istip, 
parentClade)
  for i in children
recursiveBuild(i, cladeArray, currentClade, current)
  end
end

If I use this function, the cladeArray array is properly fleshed out with 
the types generated from the read in information. This type is the 
PhyXElement type as can be seen from the constructor, it takes 4 argument 
and the type contains 4 variables.

However PhyXElement can be extended to contain more than those 4 variables. 
So I modified the above function with two macros that can alter the code 
accordingly. It is below, but not since they are not provided any 
arguments, the result should be functionally identical to the above 
definition:

macro gatherPhyXMLAttributes(attributes...)
  code = :(begin end)
  for i in attributes
push!(code.args, :($(symbol($i = readPhyXML(xml, ::$i)
  end
  code
end


macro makePhyXElement(attributes...)
  line = cladeArray[currentClade.nodeIndex] = PhyXElement(label, isroot, 
istip, parentClade
  for i in attributes
line = $(line), $i
  end
  :($(symbol($(line)
end


function recursiveBuild(xmlclade, cladeArray, currentClade, parentClade::Int
)
  # Update the node tracker.
  currentClade.nodeIndex += 1
  current = currentClade.nodeIndex # Initialize a local variable called 
current, taken from the currentClade variable to keep as the variable to 
pass to furthur recursive calls as the parent index.
  # Get name of clade element.
  label = 
  isroot = currentClade == 0 ? true : false
  children = get_elements_by_tagname(xmlclade, clade)
  istip = length(children) == 0 ? true : false
  # Get and process all additional data
  @gatherPhyXMLAttributes
  # Make the clade object by correctly calling the constructor.
  @makePhyXElement
  for i in children
recursiveBuild(i, cladeArray, currentClade, current)
  end
end

If I now expand the function definition using macro expand the function 
body is:

currentClade.nodeIndex += 1 # line 4:
  current = currentClade.nodeIndex # line 6: 
  label =  # line 7: 
  isroot = if currentClade == 0 
true 
  else  
false 
  end # line 8: 
  children = get_elements_by_tagname(xmlclade,clade) # line 9: 
  istip = if length(children) == 0 
true 
  else  
false 
  end # line 11: 
  begin  
  end # line 12: 
  cladeArray[currentClade.nodeIndex] = PhyXElement(label, isroot, 
istip,parentClade
) # line 13: 
  for i = children # line 14: 
recursiveBuild(i,cladeArray,currentClade,current) 
  end


So what has happened is the first macro made the rather useless begin end 
lines, which would contain additional operations if the macro received 
arguments. The second macro without arguments recreates the call to the 
PhyXElement constructor. However if I run this version of the function, the 
cladeArray array remains an array of undefined element and is not filled in 
with PhyXElement types as it is when the previous version of the function 
is used. I wondered if the issue is to do with macro hygiene but I can't 
see it. I hope someone can look at how I've written this and see if I've 
made a derp - Arrays are mutable so I figured cladeArray should end up the 
same as it did when the previous version of the function is used. 

Thanks,
Ben.
 

[julia-users] Equivalent to R's seq()?

[Ben Ward]

Hi all,

In R if I want to generate a sequence of numbers, say 5, 10, 15, 20, in R 
seq(from=5, to=20, by=5) would achieve this. I've had a glance over the 
sections of the standard library functions doc and didn't see a function 
like this - (unless it has a totally different name or a package provides 
it). Is there a function like this for Julia that does this easily?

Thanks,
Ben.

[julia-users] Composite type fields and types defined

[Ben Ward]

Hi,

Is there a way to extract the names and types of the fields in a given 
composite type? 

Say:

type MyType
 a::Int
end

Is there a way to get the field 'a', and the type of the field Int? I think 
I've seen a way to get the type in the manual but not the name assigned to 
the field.

And also how do I list all the types defined in a module? I'm planning on 
attempting to use Julia's metaprogramming features for a task where I need 
to know these things.

Thanks,
Ben.

Re: [julia-users] Composite type fields and types defined

[Ben Ward]

Thanks Mauro!

On Saturday, March 1, 2014 8:40:31 AM UTC, Mauro wrote:

 have a look at 
 http://docs.julialang.org/en/release-0.2/stdlib/base/#reflection 

 This works 
 MyType.names 
 MyType.types 

 there is also the names function.  Don't know if there is a 
 corresponding 'types' function.  Incidentally names also list all names 
 defined in a module. 

 On Sat, 2014-03-01 at 08:27, axolotl...@gmail.com javascript: wrote: 
  Hi, 
  
  Is there a way to extract the names and types of the fields in a given 
  composite type? 
  
  Say: 
  
  type MyType 
   a::Int 
  end 
  
  Is there a way to get the field 'a', and the type of the field Int? I 
 think 
  I've seen a way to get the type in the manual but not the name assigned 
 to 
  the field. 
  
  And also how do I list all the types defined in a module? I'm planning 
 on 
  attempting to use Julia's metaprogramming features for a task where I 
 need 
  to know these things. 
  
  Thanks, 
  Ben. 

[julia-users] Metaprogramming and Typedefs

[Ben Ward]

Hi,

I'm looking to make a flexible type which may or may not contain certain 
fields, as well as possibly the addition of fields later on i.e. it's final 
definition is decided at compilation. I figured meta-programming and macros 
might be the way to achieve this as well as be educational for me.

Lets have a toy example:

type MyType
 value1::Int
end

Is the first form of the type, but for another application I might want 

type MyType
value1::Int
value2::Bool
value3::Float
end

Is it possible with a macro or some form of metaprogramming to dynamically 
build up the definition, field by field - perhaps with a loop building a 
code block that is then evaluated?

I started with 

macro PXdefine(x...)

end

where x... could be a varargs I guess like (:Int, :Bool, :Float),
then a loop build those lines along the lines of

for i in 1:length(x)
:(value$i::$x[i])
end

Then the result might be able to go in between the lines type 'MyType' and 
'end'

But really I'm not 100% sure how to go about building up a chunk of code 
like this to form a more complex whole. The example I've seen in the doc's 
on code generation have evaluations inside a loop to define a function for 
many operators, but I've not seen many examples that build up expressions 
from smaller parts and then evaluate them.

Thanks,
Ben.

[julia-users] Maintaining a Julia Package - changes?

[Ben Ward]

Hi, 

I've been working on an R project as part of a PhD project for a long time 
and so have only recently returned to take a look at the Phylogenetics.jl 
package I started to create a while ago. Reading the documentation on 
packages I see what look to me like changes, like publishing checking out 
branches and such all from within Julia, this is somewhat different to the 
set up I have: I had a repository on github and on my macbook in a folder 
on my desktop, and this was the actual repo and when I used julia and 
tested the package this folder was ln -s into the .julia folder, and my 
METADATA was similarly linked into .julia. I then made commits to these 
with the github gui for osx and just pushed changes and switched branches 
as I needed. Is this kind of arrangement still acceptable or is there a 
better way of updating your package and working with updating METADATA now? 
Just think maybe I need to change my setup to avoid problems.

Thanks,
Ben.  

[julia-users] Re: Maintaining a Julia Package - changes?

[Ben Ward]

As I expected I'm having issues because I haven't kept up, for example, I 
can't update my version from 0.0.0 to 0.0.1 after a contributor merged 
changes, and Pkg.publish always returns:

*ERROR: METADATA is behind origin/metadata-v2 – run `Pkg.update()` before 
publishing*

* in publish at pkg/entry.jl:306*

* in anonymous at pkg/dir.jl:25*

* in cd at file.jl:22*

* in cd at pkg/dir.jl:25*

* in publish at pkg.jl:57*

Even though I've already done Pkg.update(). I'd rather not have to delete 
my repo and start again from Pkg.generate() again.

Best,
Ben.

On Friday, January 17, 2014 10:06:12 PM UTC, Ben Ward wrote:

 Hi, 

 I've been working on an R project as part of a PhD project for a long time 
 and so have only recently returned to take a look at the Phylogenetics.jl 
 package I started to create a while ago. Reading the documentation on 
 packages I see what look to me like changes, like publishing checking out 
 branches and such all from within Julia, this is somewhat different to the 
 set up I have: I had a repository on github and on my macbook in a folder 
 on my desktop, and this was the actual repo and when I used julia and 
 tested the package this folder was ln -s into the .julia folder, and my 
 METADATA was similarly linked into .julia. I then made commits to these 
 with the github gui for osx and just pushed changes and switched branches 
 as I needed. Is this kind of arrangement still acceptable or is there a 
 better way of updating your package and working with updating METADATA now? 
 Just think maybe I need to change my setup to avoid problems.

 Thanks,
 Ben.  

Re: [julia-users] Re: Maintaining a Julia Package - changes?

[Ben Ward]

My version info is:

Julia Version 0.3.0-prerelease+1043

Commit ab37fde* (2014-01-17 10:38 UTC)

Platform Info:

  System: Darwin (x86_64-apple-darwin13.0.0)

  CPU: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz

  WORD_SIZE: 64

  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)

  LAPACK: libopenblas

  LIBM: libopenlibm



On Saturday, January 18, 2014 3:46:55 AM UTC, Isaiah wrote:

 Strange. Well you could push manually and make a pull request against 
 METADATA. What version of Julia are you running? (`versioninfo()`)

 On Fri, Jan 17, 2014 at 10:33 PM, Ben Ward axolotl...@gmail.comjavascript:
  wrote:

 I've just removed my packages .julia folder and redid Pkg.init() so I 
 should have new .julia/METADATA. I made a minor update to the read me of 
 Phylogenetics.jl's read-me, and done Pkg.tag() to make the registered 
 version 0.0.1 instead of 0.0.0 on account of some welcome changes and bug 
 fixes by a contributor. Now Pkg.publish() says:

 *julia **Pkg.publish()*

 *ERROR: METADATA is behind origin/metadata-v2 – run `Pkg.update()` before 
 publishing*

 * in publish at pkg/entry.jl:306*

 * in anonymous at pkg/dir.jl:25*

 * in cd at file.jl:22*

 * in cd at pkg/dir.jl:25*

 * in publish at pkg.jl:57*

 The git log:

 cd .julia/METADATA
 git log

 n95753:METADATA wardb$ git log

 commit eef97d0d9c0bacad25507b36624ad5b7652ef57a

 Author: ward9250 ben.w...@sainsbury-lab.ac.uk

 Date:   Sat Jan 18 03:31:04 2014 +



 Tag Phylogenetics v0.0.1



 commit bfae06320259898aa056026352905a126f63521d

 Author: Dahua Lin linda...@gmail.com

 Date:   Fri Jan 17 16:55:12 2014 -0600



 Tag StatsBase v0.3.4



 commit c389e0e9e6cdf7f8004a958e04c36d10371c6df1

 Author: Steven G. Johnson stev...@mit.edu

 Date:   Fri Jan 17 15:58:16 2014 -0500



 Tag PyCall v0.4.0



 commit f1aed389a89712589e3c2141c6f2988b17e9b4d9

 Merge: bbb9754 8418f37

 Author: Stefan Karpinski ste...@karpinski.org

 Date:   Fri Jan 17 10:00:42 2014 -0800



 Merge pull request #517 from jverzani/pull-request/8418f379

 

 commit bfae06320259898aa056026352905a126f63521d

 Author: Dahua Lin linda...@gmail.com

 Date:   Fri Jan 17 16:55:12 2014 -0600



 Tag StatsBase v0.3.4



 commit c389e0e9e6cdf7f8004a958e04c36d10371c6df1

 Author: Steven G. Johnson stev...@mit.edu

 Date:   Fri Jan 17 15:58:16 2014 -0500



 commit eef97d0d9c0bacad25507b36624ad5b7652ef57a

 Author: ward9250 ben.w...@sainsbury-lab.ac.uk

 Date:   Sat Jan 18 03:31:04 2014 +



 Tag Phylogenetics v0.0.1



 commit bfae06320259898aa056026352905a126f63521d

 Author: Dahua Lin linda...@gmail.com

 Date:   Fri Jan 17 16:55:12 2014 -0600



 Tag StatsBase v0.3.4



 commit c389e0e9e6cdf7f8004a958e04c36d10371c6df1

 Author: Steven G. Johnson stev...@mit.edu

 Date:   Fri Jan 17 15:58:16 2014 -0500



 Tag PyCall v0.4.0



 commit f1aed389a89712589e3c2141c6f2988b17e9b4d9

 Merge: bbb9754 8418f37

 Author: Stefan Karpinski ste...@karpinski.org

 Date:   Fri Jan 17 10:00:42 2014 -0800



 :



 On Friday, January 17, 2014 10:06:12 PM UTC, Ben Ward wrote:

 Hi, 

 I've been working on an R project as part of a PhD project for a long 
 time and so have only recently returned to take a look at the 
 Phylogenetics.jl package I started to create a while ago. Reading the 
 documentation on packages I see what look to me like changes, like 
 publishing checking out branches and such all from within Julia, this is 
 somewhat different to the set up I have: I had a repository on github and 
 on my macbook in a folder on my desktop, and this was the actual repo and 
 when I used julia and tested the package this folder was ln -s into the 
 .julia folder, and my METADATA was similarly linked into .julia. I then 
 made commits to these with the github gui for osx and just pushed changes 
 and switched branches as I needed. Is this kind of arrangement still 
 acceptable or is there a better way of updating your package and working 
 with updating METADATA now? Just think maybe I need to change my setup to 
 avoid problems.

 Thanks,
 Ben.