Re: [julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?
http://docs.julialang.org/en/latest/manual/faq/#how-do-abstract-or-ambiguous-fields-in-types-interact-with-the-compiler
http://docs.julialang.org/en/latest/manual/faq/#how-should-i-declare-abstract-container-type-fields
--Tim
On Wednesday, June 10, 2015 10:53:44 PM Steven G. Johnson wrote:
On Thursday, June 11, 2015 at 6:21:01 AM UTC+2, Lyndon White wrote:
Containers of abstract types are not performant,
However containers of concrete types are not generic.
That's why you should use parameterized types like Matrix{T}. This gives
you an abstract family (Matrix) of concrete types (Matrix{T}), giving
mostly the best of both worlds (generic code + performance).
Re: [julia-users] Current Performance w Trunk Compared to 0.3
I've seen the same. Looked away for a few weeks, and my code got ~5x slower. There's a lot going on so it's hard to say without detailed testing. However this code was always very sensitive to optimization to be able to specialize code which read data of different types. I got massive increases in memory allocations. I'll try to narrow it down, but it seems like perhaps something was done with optimization passes or type inference? On Wednesday, June 10, 2015 at 9:31:59 AM UTC-4, Kevin Squire wrote: Short answer: no, poor performance across the board is not a known issue. Just curious, do you see these timing issues locally as well? In other words, is it a problem with Julia, or a problem with Travis (the continuous integration framework)? It might be the case that some changes in v0.4 have (possibly inadvertantly) slowed down certain workflows compared with v0.3, whereas others are unchanged or even faster. Could you run profiling and see what parts of the code are the slowest, and then file issues for any slowdowns, with (preferably minimal) examples? Cheers, Kevin On Wed, Jun 10, 2015 at 9:10 AM, andrew cooke and...@acooke.org javascript: wrote: Is it the current poor performance / allocation a known issue? I don't know how long this has been going on, and searching for performance in issues gives a lot of hits, but I've been maintaining some old projects and noticed that timed tests are running significant;y slower with trunk than 0.3. CRC.jl was 40x slower - I ended up cancelling the Travis build, and assumed it was a weird glitch that would be fixed. But now I am seeing slowdowns with IntModN.jl too (factor more like 4x as slow). You can see this at https://travis-ci.org/andrewcooke/IntModN.jl (compare the timing results in the two jobs) and at https://travis-ci.org/andrewcooke/CRC.jl/builds/66140801 (i have been cancelling jobs there, so the examples aren't as complete). Andrew
[julia-users] Method parameters and union types
I want the following function
function t_or_void{T}(::Type{Union(T, Void)})
return T
end
to work like this:
julia t_or_void(Union(Int, Void))
Int64
But in reality, it does this:
julia t_or_void(Union(Int, Void))
UnionType
Is there a way to make this work, or a way to extract T from Union(T,
Void)? The 'Void' isn't special -- it could be any determinate type.
Thanks,
D
Re: [julia-users] Method parameters and union types
Are you looking for the type-intersection perhaps?
julia typeintersect(Union(Int,Float64), Float64)
Float64
On Thu, Jun 11, 2015 at 3:15 PM David Gold david.gol...@gmail.com wrote:
I want the following function
function t_or_void{T}(::Type{Union(T, Void)})
return T
end
to work like this:
julia t_or_void(Union(Int, Void))
Int64
But in reality, it does this:
julia t_or_void(Union(Int, Void))
UnionType
Is there a way to make this work, or a way to extract T from Union(T,
Void)? The 'Void' isn't special -- it could be any determinate type.
Thanks,
D
Re: [julia-users] Method parameters and union types
Just the opposite =p I don't know what T is, but I'd like to be able work
with it. For instance, if somebody gives me an Array{Union(T, U)}, where I
know U but not T, I'd like to be able to return an Array{T}.
On Thursday, June 11, 2015 at 3:29:02 PM UTC-4, Jameson wrote:
Are you looking for the type-intersection perhaps?
julia typeintersect(Union(Int,Float64), Float64)
Float64
On Thu, Jun 11, 2015 at 3:15 PM David Gold david@gmail.com
javascript: wrote:
I want the following function
function t_or_void{T}(::Type{Union(T, Void)})
return T
end
to work like this:
julia t_or_void(Union(Int, Void))
Int64
But in reality, it does this:
julia t_or_void(Union(Int, Void))
UnionType
Is there a way to make this work, or a way to extract T from Union(T,
Void)? The 'Void' isn't special -- it could be any determinate type.
Thanks,
D
[julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?
i don't know of any docs that justify this, but i would assume that Union
and Number would have very similar performance.
also, this isn't as bad as you may think. it depends a lot on what you are
doing. if you're array crunching, yes, it's a big deal, because you have
to look at type tags all the time. but i you're just storing data at a
high level and then passing it to other routines, those routines may still
be specialised.
for example:
julia foo(a::Int) = (println(int); a)
foo (generic function with 1 method)
julia foo(a::Number) = (println(number); a)
foo (generic function with 2 methods)
julia map(foo, Number[1,1.0])
int
number
2-element Array{Real,1}:
1
1.0
you can see that the specialised, efficient, foo() is called. and if that
is where most of your work is done, you are fine.
so never is a bit strong. sometimes abstract types can be very useful.
andrew
On Thursday, 11 June 2015 01:21:01 UTC-3, Lyndon White wrote:
Containers of abstract types are not performant,
This is documented in the performance notes section of the manual.
However containers of concrete types are not generic.
For example, I am currently redoing a large section of code from using
Float64s to using Numbers, so that I can use DualNumbers to check
derivatives.
The reason Number (and other abstract types) are slow, is because are full
of pointers.
Since each element of a Matix{Number} could potentially be of a different
type, eg some could be Int8s and some could be Float64s.
But in practice the varse majority of the time, every element is of the
same type.
The 3 signitures:
Matix{Number}
Matrix{Union(Number, Float64, Float32, Float16, Int...) }
Matrix{T:Number}
All can contain the same information, as they will all contain a
Matrix{Number}, and none of them will contain a Matrix{NotNumber},
(Though to fit a concrete typed matrix into a matrix of abstract type you
need to run a convert),
But they have very different performance.
As I can show with the following Notebook:
http://nbviewer.ipython.org/gist/oxinabox/a00d3b1eb5584467e6b7
Which also compares with Any just for the sake of seeing if contrete types
help (they don't)
What can be seen t Matrix{Union(Number, Float64) }, Matrix{T:Number} and
Any perform about the same
and that Matrix{Number} is 3 orders of magnitude slower (not unexpected).
Ergo, because the former can contain anything that can be contained in the
Matrix{Number}, there is AFAICT, no reason not to use them.
Worse case senario, the contents really is a Matrix of mixed numeric types
and the performance falls back to the Matrix{Number} case.
Syntactically perhaps it is less nice.
I have a type alias for Matrix{Union(Number, Float64,...)} which solves
that, to an extent.
Argument could be made that Matrix{T:Number} is even better, but it is
annoying to refactor code to use that as it is not a simple find and
replace as a type parameter needs to be added to all functions.
Thoughts, comments,
Reasons why Matrix{Number} might be users over either of the other cases?
Re: [julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?
The rule of thumb is this:
Using abstract types to describe *behaviors* is fine, using them for
*locations* is bad.
A field is a location, the element type of an array is a set of locations.
If you want these to be fast, they should be concretely typed. The types in
a method signature describes a behavior, not a location, so it's fine for
the types to be abstract.
On Thu, Jun 11, 2015 at 9:26 AM, andrew cooke and...@acooke.org wrote:
i don't know of any docs that justify this, but i would assume that Union
and Number would have very similar performance.
also, this isn't as bad as you may think. it depends a lot on what you
are doing. if you're array crunching, yes, it's a big deal, because you
have to look at type tags all the time. but i you're just storing data at
a high level and then passing it to other routines, those routines may
still be specialised.
for example:
julia foo(a::Int) = (println(int); a)
foo (generic function with 1 method)
julia foo(a::Number) = (println(number); a)
foo (generic function with 2 methods)
julia map(foo, Number[1,1.0])
int
number
2-element Array{Real,1}:
1
1.0
you can see that the specialised, efficient, foo() is called. and if that
is where most of your work is done, you are fine.
so never is a bit strong. sometimes abstract types can be very useful.
andrew
On Thursday, 11 June 2015 01:21:01 UTC-3, Lyndon White wrote:
Containers of abstract types are not performant,
This is documented in the performance notes section of the manual.
However containers of concrete types are not generic.
For example, I am currently redoing a large section of code from using
Float64s to using Numbers, so that I can use DualNumbers to check
derivatives.
The reason Number (and other abstract types) are slow, is because are
full of pointers.
Since each element of a Matix{Number} could potentially be of a different
type, eg some could be Int8s and some could be Float64s.
But in practice the varse majority of the time, every element is of the
same type.
The 3 signitures:
Matix{Number}
Matrix{Union(Number, Float64, Float32, Float16, Int...) }
Matrix{T:Number}
All can contain the same information, as they will all contain a
Matrix{Number}, and none of them will contain a Matrix{NotNumber},
(Though to fit a concrete typed matrix into a matrix of abstract type
you need to run a convert),
But they have very different performance.
As I can show with the following Notebook:
http://nbviewer.ipython.org/gist/oxinabox/a00d3b1eb5584467e6b7
Which also compares with Any just for the sake of seeing if contrete
types help (they don't)
What can be seen t Matrix{Union(Number, Float64) }, Matrix{T:Number}
and Any perform about the same
and that Matrix{Number} is 3 orders of magnitude slower (not unexpected).
Ergo, because the former can contain anything that can be contained in
the Matrix{Number}, there is AFAICT, no reason not to use them.
Worse case senario, the contents really is a Matrix of mixed numeric
types and the performance falls back to the Matrix{Number} case.
Syntactically perhaps it is less nice.
I have a type alias for Matrix{Union(Number, Float64,...)} which solves
that, to an extent.
Argument could be made that Matrix{T:Number} is even better, but it is
annoying to refactor code to use that as it is not a simple find and
replace as a type parameter needs to be added to all functions.
Thoughts, comments,
Reasons why Matrix{Number} might be users over either of the other cases?
Re: [julia-users] Re: Organizing code around Jump models
There's a lot going on here, but a few observations:
- In the (unreleased) development version of JuMP, we store a dictionary of
variables used in the model (https://github.com/JuliaOpt/JuMP.jl/pull/446),
so you won't need to manually keep track of this e.g., with the var
dictionary above.
- That said, there's a bit of a performance penalty for going through
dictionaries. Don't worry about this until it becomes a bottleneck, though.
- Abstract types specify what do can do with a certain object, but not
what fields are available in an object. So it's a bit of poor style to
access the fields of an object in generic code (in setLineCapa!, you access
m.mip, but m is not guaranteed to have this field). I admit to doing this
myself sometimes, though. The suggested workaround is to define accessor
methods like mip(m::Model) = m.mip. Then the delegation/proxy pattern
above is replaced by defining new methods, e.g., mip(m::Model) = m.uc.mip.
Does this help? It's always a challange to design good abstractions, and
we're definitely open to making it easier to build modular models like this
in JuMP. Are there Julia language features which you feel are making this
more difficult than you expect?
Best,
Miles
On Thu, Jun 4, 2015 at 10:28 PM, Francois Gilbert
francois.gilbert...@gmail.com wrote:
Hi Miles,
It would be easier for me to give some excerpts of the Julia sources I am
working with. I think it would suffice to illustrate the mechanic we are
trying to implement.
Say we have two abstract types:
abstract BasicUC
abstract CCModel:BasicUC
These are declared in separate modules. BasicUC stands for basic
unit-commitment and types that derive it have access to functionalities
such as:
function setLineCapa!(m::BasicUC)
JuMP.@addConstraint(m.mip, _contraint1[l = keys(m.net.branch), t
= m.timeIndex],
m.var[:flow][l,t] = m.net.branch[l].Fmax)
...
end
CCModel stands for an extended unit-commitment formulation, where robust
reserve requirement are implemented. Details are irrelevant. A type that
derives CCModel has access to, say:
function setRobustReserve!(m::CCModel)
...
end
Extending the functionality specific to BasicUC - let a type “Model” such
that
type Model{T:BasicUC} : CCModel
uc# Hook to BasicUC
# Proxies to BasicUC:
demand # exogenous stuff
mip # JuMP model
. . .
function Model(net,scenario)
this=new()
this.uc=T(net,scenario)
. . .
# Delegation to BasicUC
this.demand=this.uc.demand
this.mip=this.uc.mip
this.var=this.uc.var
. . .
# Specific to CCModel
this.contingencies=scenario[:contingencies]
JuMP.@defVar(this.mip, _post_gen[keys(this.net.gen),
keys(this.contingencies),
this.timeIndex])
this.var[:post_gen]= _post_gen
. . .
return this
end
end
Now, let a concrete type BasicModel be defined in, say module
BasicUnitCommitment, and such that :
BasicModel:BasicUC
Type Model{T} above can then be instantiated and tailored, with:
m=Model{BasicUnitCommitment.BaseModel}(net,scenario)
# from the base model (BasicUC)
setBalanceEq!(m)
setLineCapa!(m)
setInitRamping!(m)
# these are overloaded by CCModel
setObjective!(m)
setGenCapa!(m)
setRamping!(m)
# these are specific to CCModel
setPostBalanceEq!(m)
setRobustReserve!(m)
. . .
So that's the idea. Perhaps I am overdoing this, but I think having some
kind of encapsulation is essential for developing larger applications.
Emulating inheritance, as is done above, is rather straightforward, but
perhaps a little laborious, and there might be a better way to go about
it.
François
Le mercredi 3 juin 2015 16:12:21 UTC-5, Miles Lubin a écrit :
Hi Francois,
Could you give an example of how you might organize this code in a more
namespace heavy language like C++ or Java? I think this is quite a general
question on how to structure large Julia applications, not really specific
to JuMP.
Best,
Miles
On Wednesday, June 3, 2015 at 9:07:03 PM UTC+2, Francois Gilbert wrote:
Hi all,
Is anyone aware of any effort building large OR applications? I am
currently working at organizing a relatively large code base making use of
the JuMP libraries. The context is unit-commitment and transmission
planning. Flexibility and code re-use is among our priorities.
Here are some of the design choices we made so far:
1) JuMP models are encapsulated in types that hold:
a) references to the variable names, constraints (when multipliers are
needed) and indices. These are otherwise difficult to extract directly
from the JuMP object
b) data structure that are required for model specification (generator
characteristics, transmission lines, etc)
c) exogenous variable values (demand scenarios,
Re: [julia-users] Error with Pkg.add(Cairo)
More people are having issues... https://github.com/JuliaLang/Homebrew.jl/issues/85 https://github.com/JuliaLang/Cairo.jl/issues/105
Re: [julia-users] Error with Pkg.add(Cairo)
Could you please Pkg.update() and try again, I believe I have deployed a fix. -E On Thu, Jun 11, 2015 at 1:39 PM, Stepa Solntsev solnt...@gmail.com wrote: More people are having issues... https://github.com/JuliaLang/Homebrew.jl/issues/85 https://github.com/JuliaLang/Cairo.jl/issues/105
[julia-users] Re: Method parameters and union types
On Friday, June 12, 2015 at 5:15:54 AM UTC+10, David Gold wrote:
I want the following function
function t_or_void{T}(::Type{Union(T, Void)})
return T
end
to work like this:
julia t_or_void(Union(Int, Void))
Int64
Works like this on Julia 3.8, seems to be a regression?
But in reality, it does this:
julia t_or_void(Union(Int, Void))
UnionType
Is there a way to make this work, or a way to extract T from Union(T,
Void)? The 'Void' isn't special -- it could be any determinate type.
Thanks,
D
[julia-users] Re: Atom package
Looking into this more, it looks like the repo can have any name you want. The important thing is that the package is named language-julia, not the repo. On Thursday, June 11, 2015 at 8:05:20 PM UTC-7, Spencer Lyon wrote: Good to see something for atom-language-julia. That is what the package was named originally, but it wasn't maintained so it was forked and renamed language-julia. I don't think there will really be any reusable components. I just dont know another editor/tool that uses textmate style grammar in cson format or atom's snippet syntax, for example.
[julia-users] Re: Atom package
Ahh that's a great distinction-- good catch. Knowing that I'd vote that the repo be named JuliaLang/atom-language-julia // Spencer On June 11, 2015 at 9:49:28 PM MST, John Myles White johnmyleswh...@gmail.com wrote:Looking into this more, it looks like the repo can have any name you want. The important thing is that the package is named language-julia, not the repo.On Thursday, June 11, 2015 at 8:05:20 PM UTC-7, Spencer Lyon wrote:Good to see something for atom-language-julia. That is what the package was named originally, but it wasn't maintained so it was forked and renamed language-julia. I don't think there will really be any reusable components. I just dont know another editor/tool that uses textmate style grammar in cson format or atom's snippet syntax, for example.
[julia-users] Re: JuliaCon registrations open
On Tuesday, 9 June 2015 22:32:31 UTC+2, Viral Shah wrote: Last day to book the hotel at the discounted rate - for anyone procrastinating! I would love to attend, but I can't make the trip across the ocean. I hope to see a JuliaCon in Europe some day! Cheers, Daniel.
[julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?
Oh, wait you meant that Matrix{Union(...)} should have same performance as
Matrix{Number}?
Yes, I also assume that (I guess I should test).
But Union{Matrix{Float64}, Matrix{Number},...}
is fast.
---
On Friday, 12 June 2015 08:47:12 UTC+8, Lyndon White wrote:
On Thursday, 11 June 2015 21:26:25 UTC+8, andrew cooke wrote:
i don't know of any docs that justify this, but i would assume that Union
and Number would have very similar performance.
You would be wrong.
Union and {T:Number} have similar performance.
Number performed 1000x worse.
For several reasons {T:Number} is better, inclusing more sane error
messages, less having to type out all subtypes etc.
[julia-users] Re: Atom package
Good to see something for atom-language-julia. That is what the package was named originally, but it wasn't maintained so it was forked and renamed language-julia. I don't think there will really be any reusable components. I just dont know another editor/tool that uses textmate style grammar in cson format or atom's snippet syntax, for example.
[julia-users] Re: Atom package
I see there's at least some precedent https://github.com/zargony/atom-language-rust for atom-language-julia maybe? Would this plugin be entirely atom-specific, or could other editors also reuse the same code? On Wednesday, June 10, 2015 at 3:41:08 PM UTC-7, John Myles White wrote: It is a very strong convention, though. -- John On Wednesday, June 10, 2015 at 1:52:01 PM UTC-7, Spencer Lyon wrote: Definitely not a requirement, just convention. If there are other proposals out there I'm happy to entertain them. On Wednesday, June 10, 2015 at 1:51:07 PM UTC-7, Tony Kelman wrote: Does atom require you to name the repository language-julia? I'm generally in favor of your plan here but maybe a more specific repo name would be clearer, if that's possible? On Wednesday, June 10, 2015 at 10:33:45 AM UTC-7, Spencer Lyon wrote: Right now the atom package for adding syntax highlighting for Julia is https://github.com/tpoisot/language-julia Based on issues like this one ( https://github.com/tpoisot/language-julia/issues/8) it seems like the grammar.cson is incorrect. I did a fresh restart of the grammar based on the official text mate bundle https://github.com/JuliaLang/Julia.tmbundle and all these issues seem to be resolved. It is a relief to not have syntax highlighting treat everything as a string mid-file! I've also added everything in Base.REPLCompletions.latex_symbols and Base.REPLCompletions.emoji_symbols as snippets. I'm writing here to propose two things: 1. Contact the maintainer of the current language-julia atom package, merge in the new features he has added, and begin using the new one generated from the tmBundle as the official Julia support for atom 2. To make it seem more official I would like house the repository under the JuliaLang github organization in a repo JuliaLang/language-julia. What do people think about this idea?
Re: [julia-users] Error with Pkg.add(Cairo)
Works like a charm. Thank you! On Thu, Jun 11, 2015 at 5:17 PM, Elliot Saba staticfl...@gmail.com wrote: Could you please Pkg.update() and try again, I believe I have deployed a fix. -E On Thu, Jun 11, 2015 at 1:39 PM, Stepa Solntsev solnt...@gmail.com wrote: More people are having issues... https://github.com/JuliaLang/Homebrew.jl/issues/85 https://github.com/JuliaLang/Cairo.jl/issues/105
[julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?
On Thursday, 11 June 2015 21:26:25 UTC+8, andrew cooke wrote:
i don't know of any docs that justify this, but i would assume that Union
and Number would have very similar performance.
You would be wrong.
Union and {T:Number} have similar performance.
Number performed 1000x worse.
For several reasons {T:Number} is better, inclusing more sane error
messages, less having to type out all subtypes etc.
