[julia-users] Re: Packing and unpacking parameters
I've struggled with this exact problem in python in the past (e.g. https://stackoverflow.com/questions/16182898/unpacking-parameters-for-a-simulation). It's exacerbated by the fact that interfaces to solvers, optimizers, etc. often require the parameters be passed in as a vector, so using dictionary won't help. Dictionary access in tight loops also has obviously bad performance implications. My intuition is that a macro is the way out, something like: @unpack params a1 a2 a3 b1 b2 C # etc . . . as you suggest. The tricky bit is implementing it in a way that avoids unnecessary memory allocation. I'm not sure how good the LLVM optimization passes are but I suspect that doing it as you suggested previously (e.g. translating the above into a1 = params[1], a2 = params[2], etc.) will result in unnecessarily copying things onto the stack from the params array. —James On Friday, March 7, 2014 8:15:58 AM UTC-6, Yuuki Soho wrote: It's a bit of a stupid question, but I don't really know how to deal with this efficiently. So, in many application I have some model with parameters, and I want to the able to change the number of parameters, or they order easily. For passing parameters to functions I want to pack them into a vector p, such that I don't have huge function definition, but inside the function's body I'd prefer to have all the parameters given by their name, so I can use them in equations (instead of using p[1], p[2], ...). I can write two functions p = pack(a,b,c) and (a,b,c) = unpack(p) but that's pretty restrictive because if you add or remove a parameters, I have to change all my function calls and definition. If I add another model I also need to write another pack and unpack pairs. Is there an better approach to do this in Julia ? I was thinking maybe doing a macro @unpack p that would spawn all the variables needed, but I'm not sure that's the right way to do it.
[julia-users] Re: Packing and unpacking parameters
Have you looked into using keyword arguments? http://julia.readthedocs.org/en/latest/manual/functions/#keyword-arguments On Friday, March 7, 2014 8:15:58 AM UTC-6, Yuuki Soho wrote: It's a bit of a stupid question, but I don't really know how to deal with this efficiently. So, in many application I have some model with parameters, and I want to the able to change the number of parameters, or they order easily. For passing parameters to functions I want to pack them into a vector p, such that I don't have huge function definition, but inside the function's body I'd prefer to have all the parameters given by their name, so I can use them in equations (instead of using p[1], p[2], ...). I can write two functions p = pack(a,b,c) and (a,b,c) = unpack(p) but that's pretty restrictive because if you add or remove a parameters, I have to change all my function calls and definition. If I add another model I also need to write another pack and unpack pairs. Is there an better approach to do this in Julia ? I was thinking maybe doing a macro @unpack p that would spawn all the variables needed, but I'm not sure that's the right way to do it.
[julia-users] Re: Packing and unpacking parameters
Basically I want to define only once the name of my parameters, and never write them down in function's arguments or declare them in function's body, such that I can add, remove, rename, reorder parameters without having to do anything, and still have all my parameters available in my functions body. I was thinking of something like this: I define a list of variables name n = [a,b,c] in my main function and then having a macro @unpack n p that would generate: a = p[1] b = p[2] c = p[2] Such that I can do something like : function F1(n,p) @unpack n p return a*cos(b-c) end Does that make any sense, or I am missing something obvious ?
[julia-users] Re: Packing and unpacking parameters
How amazing! I have been about to post the very same question to this list!
So, I wouldn't say it's a stupid question at all :)
I believe this kind of packing/unpacking of several heterogeneous (in
structure, not in type) parameters appear a lot in modeling and
optimization.
This probably is a question transversal to many (all) programming
languages, but after porting some code of mine from Python to Julia this
issue is now hitting me as the major bottleneck.
I need to pack several parameters which are scalars, vectors and symmetric
matrices of fixed size.
My first attempt was to define a generic packing function to map from a
Dict of symbol=parameter to the vector form which used a pack function
dispatched on the parameters type. The vector form was exposed externally
to the optimization routines and the Dict form was used internally in the
model. The problem of that format approach was that Julia base types were
not expressive enough, for example, to know the size of a Vector parameter
or to know that a Matrix was symmetric and thus only a set of its unique
elements must be packed to the vector form.
I then thought about creating my own Types, but that would represent a huge
task not worth it. If it were possible to delegate methods from a type to
one of its fields (https://github.com/JuliaLang/julia/pull/3292) that would
be a much easier task:
immutable SymmetricMatrix{T}
a::Matrix{T}
end
@somehow_delegate_methods SymmetricMatrix -- SymmetricMatrix.a
I ended up defining a 'format' array made of (Symbol, Symbol) pairs were
the first symbol is the parameter name and the second symbol is just a
label indicating the type of the parameter over which some unpack/pack are
dispatched (if interested see
https://github.com/cdsousa/Robotics.jl/blob/master/src/dynparams.jl).
Anyway I still have a feeling that this is suboptimal, and I wonder what
are the common practices in other languages.
I even did a rough search on StackOverflow but found nothing so far...
On Friday, March 7, 2014 2:15:58 PM UTC, Yuuki Soho wrote:
It's a bit of a stupid question, but I don't really know how to deal with
this efficiently.
So, in many application I have some model with parameters, and I want to
the able to change the number of parameters, or they order easily.
For passing parameters to functions I want to pack them into a vector p,
such that I don't have huge function definition, but inside
the function's body I'd prefer to have all the parameters given by their
name, so I can use them in equations (instead of using p[1], p[2], ...).
I can write two functions p = pack(a,b,c) and (a,b,c) = unpack(p) but
that's pretty restrictive because if you add or remove a parameters, I have
to change all
my function calls and definition. If I add another model I also need to
write another pack and unpack pairs.
Is there an better approach to do this in Julia ? I was thinking maybe
doing a macro @unpack p that would spawn all the variables needed, but I'm
not
sure that's the right way to do it.
[julia-users] Re: Packing and unpacking parameters
RE some kind of `@unpack` macro: have a look at this question on Stack Overflow and its answers: http://stackoverflow.com/questions/9345056/in-clojure-how-to-destructure-all-the-keys-of-a-map/11182432#11182432 Interestingly enough it is possible to an extent, but for many reasons ends up being a bad idea. What you probably want instead is the Dictionary type - see below. Using it, you can write params[:a], params[:b] etc. http://docs.julialang.org/en/release-0.2/stdlib/base/#indexable-collections That said, it would be nice to have some kind of map destructuring macro similar to Clojure's. Hopefully the pattern matching libraries will eventually include something along those lines. Hope this helps, Mike
[julia-users] Re: Packing and unpacking parameters
Maybe I don't understand the problem well enough, but I think that an immutable type could work pretty well. You will have to qualify your references to the members, but you only have to declare the names and types once. I think it was discussed to create default constructors with keyword arguments that correspond to the field names but I'm not sure if we have it yet. Then, you could refer to the parameters by name everywhere.
[julia-users] Re: Packing and unpacking parameters
are you guys looking for something like attach in R? (it's not clear to me what you want, although a reference to python sounds like it might have been stickng values into an object's dict). andrew
[julia-users] Re: Packing and unpacking parameters
By reading your question once again, I realised that it may be not quite like mine. If you are ok with using p.a, p.b, p.c, ... then the custom composite type is the solution. A more flexible approach (as I do) is the dictionary with symbols as keys which would then be called by p[:a], p[:b], p[:c]... It is expected that in future the dot operator is going to be overloadable (https://github.com/JuliaLang/julia/issues/1974), allowing the creation of dictionary types whose symbol keys can be accessed as p.a, p.b, p.c... If you strictly want to call parameters by a, b, c... then maybe some macro can solve it, about this I cannot help. On the other hand, I really need p to be a Vector, so it can be seen as such by optimisation and linear algebra routines. In that case some packing/unpacking functions need to be called and probably some copies will always have to happen. What I'm looking for is way to have generalized and yet well performing packing/unpacking functions, flexible enough to allow easy addition/reorder of parameters. Please let me spoil a little the OP, and ask if someone can point some insights on my problem. On Friday, March 7, 2014 2:15:58 PM UTC, Yuuki Soho wrote: It's a bit of a stupid question, but I don't really know how to deal with this efficiently. So, in many application I have some model with parameters, and I want to the able to change the number of parameters, or they order easily. For passing parameters to functions I want to pack them into a vector p, such that I don't have huge function definition, but inside the function's body I'd prefer to have all the parameters given by their name, so I can use them in equations (instead of using p[1], p[2], ...). I can write two functions p = pack(a,b,c) and (a,b,c) = unpack(p) but that's pretty restrictive because if you add or remove a parameters, I have to change all my function calls and definition. If I add another model I also need to write another pack and unpack pairs. Is there an better approach to do this in Julia ? I was thinking maybe doing a macro @unpack p that would spawn all the variables needed, but I'm not sure that's the right way to do it.
