[julia-users] Re: Distributed sparse matrices?
Hi Dominique,
Madeleine Udell has a nice (unregistered?) package for parallel sparse
matrix multiplication using shared memory, which
contains a SharedSparseMatrixCSC type:
https://github.com/madeleineudell/ParallelSparseMatMul.jl
Best,
Alex.
On Friday, 22 August 2014 03:09:40 UTC+2, Dominique Orban wrote:
The Julia documentation talks about distributed arrays, but I couldn't
find anything about distributed sparse matrices (except for an MIT class
project report, but sadly without code). If there is some and I missed it,
I apologize (and I'd very much like to know where it is). At least some of
the functionality seems to be there though:
julia A = sprand(10, 10, .25);
julia dA = distribute(A);
julia dA.chunks
1x2 Array{RemoteRef,2}:
RemoteRef(2,1,1715) RemoteRef(3,1,1716)
julia dA.cuts
2-element Array{Array{Int64,1},1}:
[1,11]
[1,6,11]
julia AA = reduce(hcat, map(fetch, dA.chunks));
julia A - AA
10x10 sparse matrix with 0 Float64 entries:
I'm wondering if the above is just a fortunate side effect of
SparseMatrixCSC being a subtype of AbstractArray, or if it's indeed
intentional and is there to stay (and can be relied upon, at least for the
near future).
If it's intentional, are there more convenient ways to initialize
distributed sparse matrices?
Thanks!
[julia-users] Re: EXTREME CAUTION: Windows uninstaller
The (un)installer system has been updated and the download links have been restored. On Thu, Aug 21, 2014 at 7:58 PM, Isaiah Norton isaiah.nor...@gmail.com wrote: Please be aware that the Windows uninstaller for the 0.3-dev series and the initial 0.3 release has a dangerous bug that could cause loss of non-julia files in the install directory. * DO NOT run the uninstaller if there are non-Julia files in the install directory. * To be safe, it is recommended to remove the uninstaller.exe file. If you wish to uninstall in the future, manually removing the install directory (if you deem safe) and the Start Menu entry will accomplish the same thing. Explanation: the uninstaller will remove *all* files and directories within the install directory. For example, if the install directory is changed from the default to an existing directory, any files or subdirectories within that existing directory will be removed when the uninstaller is executed. As a precaution, we have disabled the 0.3 download link. We have a fix in progress and the download should be available again within a few hours. Isaiah
[julia-users] Printing inside function
I have probably a very simple and trivial question... I need to force
printing of a matrix inside a function for evaluation purposes. I am
totally fine with the standard printout when you call a variable from REPL.
julia a
2x2 Array{Int64,2}:
1 3
2 4
However, print and show functions both give ugly hardly readable output,
especially when dealing with floats. What is the function that prints the
output as REPL does.
Thanks a lot.
[julia-users] Re: Printing inside function
Hi Tomas,
you can try
display(a)
Best,
Alex.
On Friday, 22 August 2014 11:07:08 UTC+2, Tomas Krehlik wrote:
I have probably a very simple and trivial question... I need to force
printing of a matrix inside a function for evaluation purposes. I am
totally fine with the standard printout when you call a variable from REPL.
julia a
2x2 Array{Int64,2}:
1 3
2 4
However, print and show functions both give ugly hardly readable output,
especially when dealing with floats. What is the function that prints the
output as REPL does.
Thanks a lot.
[julia-users] Re: Printing inside function
Thanks, I knew it is going to be easy. :)
On Friday, 22 August 2014 11:11:30 UTC+2, Alex wrote:
Hi Tomas,
you can try
display(a)
Best,
Alex.
On Friday, 22 August 2014 11:07:08 UTC+2, Tomas Krehlik wrote:
I have probably a very simple and trivial question... I need to force
printing of a matrix inside a function for evaluation purposes. I am
totally fine with the standard printout when you call a variable from REPL.
julia a
2x2 Array{Int64,2}:
1 3
2 4
However, print and show functions both give ugly hardly readable output,
especially when dealing with floats. What is the function that prints the
output as REPL does.
Thanks a lot.
Re: [julia-users] BinDeps: How to test new provider?
Hi Lucas, Thanks for working on this. Currently the fact that BinDeps doesn't install arch packages is the one thing keeping AudioIO from working on pkg.julialang.org. I don't actually have an arch machine handy, but if you get it merged into BinDeps I'll be using it ASAP. On Sunday, August 17, 2014 10:36:23 AM UTC-4, Lucas Beyer wrote: Only after sending this mail did I realize that each package needs to make use of each build-system explicitly. That now makes testing obvious.
[julia-users] array element-wise bit shift?
Well, this doesn't work
all_syms = [0:3]
all_syms . 0
ERROR: `.` has no method matching .(::Array{Int64,1}, ::Int64)
--
-- Those who don't understand recursion are doomed to repeat it
[julia-users] Re: array element-wise bit shift?
Doesn't seems to be defined right now. Of course, you can just do [i 0 for i in all_syms] and it will be just as fast as a vectorized version would be. Vectorization is a convenience, though.
[julia-users] Re: functions with state?
On Friday, August 15, 2014 2:21:05 PM UTC-4, Neal Becker wrote: I'm trying to do numerical integration. I want a function that has state information. Let's say I'm trying to integrate some function F over x. In addition, F has some state function F (x, state) = do something with x and state As others have pointed out, the simplest solution here is a closure: integrate(x - F(x,state), ...) In python (and in c++), one way is to make F a class (which can have state), and overload the function call operator. Then this can be passed to the numerical integrator. This solution always amazes me, considering that Python has closures (and now C++ has lambdas), and the closure solution is vastly simpler. (In my NLopt Python interface, I often field similar questions about how to pass extra state, so I feel like there is a cultural blindness in CS these days to closures, probably stemming from historical limitations of C++ etc.)
[julia-users] Re: array element-wise bit shift?
Steven G. Johnson wrote: Doesn't seems to be defined right now. Of course, you can just do [i 0 for i in all_syms] and it will be just as fast as a vectorized version would be. Vectorization is a convenience, though. https://github.com/JuliaLang/julia/issues/8086 -- -- Those who don't understand recursion are doomed to repeat it
[julia-users] Re: Distributed sparse matrices?
Yes I know that package, but that's for shared memory machines. I'm talking
about distributed matrices.
On Friday, August 22, 2014 2:55:46 AM UTC-4, Alex wrote:
Hi Dominique,
Madeleine Udell has a nice (unregistered?) package for parallel sparse
matrix multiplication using shared memory, which
contains a SharedSparseMatrixCSC type:
https://github.com/madeleineudell/ParallelSparseMatMul.jl
Best,
Alex.
On Friday, 22 August 2014 03:09:40 UTC+2, Dominique Orban wrote:
The Julia documentation talks about distributed arrays, but I couldn't
find anything about distributed sparse matrices (except for an MIT class
project report, but sadly without code). If there is some and I missed it,
I apologize (and I'd very much like to know where it is). At least some of
the functionality seems to be there though:
julia A = sprand(10, 10, .25);
julia dA = distribute(A);
julia dA.chunks
1x2 Array{RemoteRef,2}:
RemoteRef(2,1,1715) RemoteRef(3,1,1716)
julia dA.cuts
2-element Array{Array{Int64,1},1}:
[1,11]
[1,6,11]
julia AA = reduce(hcat, map(fetch, dA.chunks));
julia A - AA
10x10 sparse matrix with 0 Float64 entries:
I'm wondering if the above is just a fortunate side effect of
SparseMatrixCSC being a subtype of AbstractArray, or if it's indeed
intentional and is there to stay (and can be relied upon, at least for the
near future).
If it's intentional, are there more convenient ways to initialize
distributed sparse matrices?
Thanks!
[julia-users] simple indexing question
Trying to translate some of my simple numpy into julia.
Probably I should just learn to stop vectorizing everything, but it's hard to
resist the allure of compact notation
julia bit_mask
2x4 Array{Int64,2}:
0 1 0 1
0 0 1 1
OK, now find all the nonzero indices in bit_mask:
julia findn (bit_mask)
([1,2,1,2],[2,3,4,4])
Now what if I wanted to select those elements that were nonzero?
julia bit_mask[*findn (bit_mask)]
ERROR: `Array{T,N}` has no method matching Array{T,N}(::Array{Int64,2},
::Int64,
::Int64)
Well, that's not the way. What is?
--
-- Those who don't understand recursion are doomed to repeat it
Re: [julia-users] why sum(abs(A)) is very slow
There is a sumabs function in Base for this reason. We'd like to eventually be able to do stream fusion to make the vectorized version as efficient as the manually fused version, but for now there's a performance gap. Note that the vectorized version is the same speed you would get in other languages where you express this in vectorized form – it's just that you can get much faster with manual loop fusion. On Thu, Aug 21, 2014 at 11:03 PM, John Myles White johnmyleswh...@gmail.com wrote: Please read http://julialang.org/blog/2013/09/fast-numeric/ — John On Aug 21, 2014, at 8:02 PM, K Leo cnbiz...@gmail.com wrote: A is a 1-dimensional array. I used to compute sum(abs(A)). But when I changed to the following, the speed increased nearly 10 fold. Why is that? sumA=0 for i=1:length(A) sumA = sumA + abs(A[i]) end
[julia-users] Re: simple indexing question
Use `find` instead of `findn` to get the logical indexes, and then you can
just:
bit_mask[find(bit_mask)]
I know you really like that space between function names and the leading
parentheses, but in this case, it's giving you extra difficulties. I
really recommend adopting the Julian style here.
But you can make this simpler. How are you generating your bit mask? If
you can create it such that it's an array of Bools instead of Ints, you can
just index with it directly:
julia mask = [false true false true; false false true true]
2x4 Array{Bool,2}:
false true false true
false false true true
julia bit_mask[mask]
4-element Array{Int64,1}:
1
1
1
1
On Friday, August 22, 2014 9:34:04 AM UTC-4, Neal Becker wrote:
Trying to translate some of my simple numpy into julia.
Probably I should just learn to stop vectorizing everything, but it's hard
to
resist the allure of compact notation
julia bit_mask
2x4 Array{Int64,2}:
0 1 0 1
0 0 1 1
OK, now find all the nonzero indices in bit_mask:
julia findn (bit_mask)
([1,2,1,2],[2,3,4,4])
Now what if I wanted to select those elements that were nonzero?
julia bit_mask[*findn (bit_mask)]
ERROR: `Array{T,N}` has no method matching Array{T,N}(::Array{Int64,2},
::Int64,
::Int64)
Well, that's not the way. What is?
--
-- Those who don't understand recursion are doomed to repeat it
Re: [julia-users] keyword argument function masking non-kwarg function
Simultaneous dispatch on positional and keyword arguments just seems nuts. Jeff and I spent a lot of time talking about that and could never come up with anything that seemed sane or comprehensible, so it ended up being what it is now. Another option would be to dispatch on positional arguments first and *then* as a separate step do some kind of dispatch on keyword arguments, implicitly made positional. On Thu, Aug 21, 2014 at 1:46 PM, Kevin Squire kevin.squ...@gmail.com wrote: Yes, that was considered, but was never implemented. Best I could find is https://github.com/JuliaLang/julia/issues/485#issuecomment-15848966. On Thu, Aug 21, 2014 at 10:01 AM, Toivo Henningsson toivo@gmail.com wrote: I believe it was considered at some point to let dispatch distinguish between functions taking keyword arguments and functions taking no keyword arguments?
Re: [julia-users] why sum(abs(A)) is very slow
Yes, that works nicely. Obviously it would be even nicer not to have to do
that :-)
On Fri, Aug 22, 2014 at 10:53 AM, Rafael Fourquet fourquet.raf...@gmail.com
wrote:
We'd like to eventually be able to do stream fusion to make the vectorized
version as efficient as the manually fused version, but for now there's a
performance gap.
It is also not too difficult to implement a fused version via iterators,
eg:
immutable iabs{X}
x::X
end
Base.start(i::iabs) = start(i.x)
Base.next(i::iabs, s) = ((v, s) = next(i.x, s); (abs(v), s))
Base.done(i::iabs, s) = done(i.x, s)
Then sum(iabs(A)) is ways faster than sum(abs(A)) (but still slightly
slower than sumabs(A)).
Re: [julia-users] why sum(abs(A)) is very slow
Could you please explain why the iterator version is so much faster? Is it
simply from avoiding temporary array allocation?
Thanks,
--Peter
On Friday, August 22, 2014 7:53:59 AM UTC-7, Rafael Fourquet wrote:
We'd like to eventually be able to do stream fusion to make the vectorized
version as efficient as the manually fused version, but for now there's a
performance gap.
It is also not too difficult to implement a fused version via iterators,
eg:
immutable iabs{X}
x::X
end
Base.start(i::iabs) = start(i.x)
Base.next(i::iabs, s) = ((v, s) = next(i.x, s); (abs(v), s))
Base.done(i::iabs, s) = done(i.x, s)
Then sum(iabs(A)) is ways faster than sum(abs(A)) (but still slightly
slower than sumabs(A)).
Re: [julia-users] why sum(abs(A)) is very slow
Obviously it would be even nicer not to have to do that :-) My naive answer is then why not make vectorized functions lazy (like iabs above, plus dimensions information) by default? Do you have links to relevant discussions?
[julia-users] Re: simple indexing question
I should explain the real problem more clearly.
I transmit a vector of symbols x[0...N-1], N denoting the time instant. Each
symbol can take on some set of values from a set S, where the size(S) is
usually
a power of 2.
I receive a vector of values over some channel, and compute the log-likelihood
that each of the received values corresponds to each of the possible symbols in
the set S. So now I have a 2-D array of LL indexed by time instant and by the
symbol number in S.
Now each symbol corresponds to some set of binary bits, and I want the LL of
the
bits. So for each time instant, I can compute the LL for bit 0=1 by adding the
LL for all symbols that have bit 0=1.
LL(bit0=1) = \sum_{all symbols s having bit0=1} (LL(s))
This is done for each time instant (dropping the time index for clarity).
So my plan was to have a mask giving the indices of all symbols having bit0=1,
all symbols having bit1=1, etc. Then somehow use this to implement the
summation.
So the input is 2D NxM where N is the length of the received signal, and M =
size(S).
The output is 2D Nxlog2(M).
[julia-users] Manipulations with abstract parametric types
Hi all,
I've come up against a couple of surprising type-manipulation issues and I'm
wondering if I'm just missing something. Hoping someone out there knows a
better way of doing these manipulations.
The issues arise when I try writing generic algorithms on abstract parametric
types that need to make decisions based on both the concrete parameters and
the concrete type. Let's start with a simple example:
abstract MyAbstractType{T}
immutable MyType{T} : MyAbstractType{T}
a::T
b::T
end
Now let's write a generic eltype method:
Base.eltype{M:MyAbstractType}(::Type{M}) = M.parameters[1]
It's a little ugly to have to use M.parameters[1] here. While I'm curious to
know whether I'm missing a simple alternative, I can live with this.
Now let's define some arithmetic operations:
(*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
(.*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
And then try it out:
julia y = MyType(3,12)
julia 2y
MyType{Int64}(6,24)
julia 2.1y
MyType{Float64}(6.301,25.203)
So far, so good. Now let's try something a little more sophisticated:
julia A = [MyType(3,12), MyType(4,7)]
2-element Array{MyType{Int64},1}:
MyType{Int64}(3,12)
MyType{Int64}(4,7)
julia 2A
2-element Array{Any,1}:
MyType{Int64}(6,24)
MyType{Int64}(8,14)
The problem here is we're getting an Array{Any,1} back. No problem you say,
let's try to help type inference out, by specifying that a
::MyType{Int}*::Float64 is a MyType{Float64}. If we were willing to write this
using concrete types, it would be easy:
Base.promote_array_type{T:Real, S}(::Type{T}, ::Type{MyType{S}}) =
MyType{promote_type(T, S)}
But if we want to just use the abstract type, then I've been unsuccessful in
avoiding this construction:
Base.promote_array_type{T:Real, M:MyAbstractType}(::Type{T}, ::Type{M}) =
eval(M.name.name){promote_type(eltype(M), T)}
Ouch! Do we really have to use eval there? This seems likely to be a runtime
bottleneck, which seems confirmed by
code_llvm(Base.promote_array_type, (Type{Float64}, Type{eltype(A)}))
Compare it against the version generated when I instead use the concrete type,
and you'll see what I mean.
Of course these issues would be easily solved by triangular dispatch or staged
functions, but I'm hoping there's another good way that exists today.
--Tim
Re: [julia-users] Backtrace doesn't show details of modules
I'd say there's probably no need to file an issue, then. I'm glad you found the root of the problem! --Tim On Friday, August 22, 2014 06:13:00 AM Martin Klein wrote: Hi Tim, as the problem seemed to be exclusive to my system I investigated further. I used as many system libraries as possible when compiling Julia instead of letting Julia's makefile download and compile the external dependencies. When I don't link Julia against my system libraries and let Julia build its external dependencies, full backtraces do work. So in conclusion, the bug doesn't seem to be caused by Julia but by one of my system libraries. At first I suspected libunwind but the problem remains with USE_SYSTEM_LIBUNWIND=0. I haven't yet investigated further to isolate the specific library which causes this problem. Since this isn't really a Julia bug I'm also hesitant to file an issue for this. Do you think it would be worthwhile to investigate further? For now, I'm happy with my Julia installation which is linked against the self-compiled dependencies. Thanks for your help, Martin Am Montag, 18. August 2014 18:32:22 UTC+2 schrieb Tim Holy: Please say it isn't so... :-( (Is there an emoticon for deeply unhappy?) But I won't shoot the messenger :-). Anyway, I'm using v0.3.0-rc4 also, but on Kubuntu 14.04. Can you file an issue, please? --Tim On Monday, August 18, 2014 08:58:12 AM Martin Klein wrote: Hm, this is strange. I'm using julia 0.3.0-rc4 and I'm getting the truncated backtrace shown above. I've compiled Julia from Git by checking out the v0.3.0-rc4 tag. This is on Debian Testing. Any ideas what is happening here? Am Montag, 18. August 2014 17:18:51 UTC+2 schrieb Tim Holy: This has been fixed in julia 0.3. --Tim On Monday, August 18, 2014 06:42:56 AM Martin Klein wrote: Hi, I have the following problem, which makes debugging of my self-written module quite difficult. When an exception is thrown inside my module, the backtrace won't include the position of the error inside my module, but only the position where I call the function in my module. The following simple example illustrates the problem: module ErrorTest export foo function foo(x) # trigger exception println(y) end end #module Then when I'm using this module in run.jl: using ErrorTest foo(5) I get the following backtrace: ERROR: y not defined in include at ./boot.jl:245 in process_options at ./client.jl:285 in _start at ./client.jl:354 in _start at /usr/local/bin/..//lib/julia/sys.so while loading /home/martin/test/run.jl, in expression starting on line 3 As you can see, the backtrace doesn't reach into the module ErrorTest, so I don't get any information in which part of ErrorTest the error occurs (i.e. line 6 in my example). For larger and complicated modules, this makes debugging nearly impossible, since I don't even get information in which function in my module the error occurs. I'm currently using v0.3-rc4. Is this a bug or intented behaviour? I couldn't find any bug report about this. If this is intended, what is your usual approach to obtain a detailed backtrace when an error occurs inside a module? Thanks, Martin
[julia-users] Re: [julia-dev] Re: EXTREME CAUTION: Windows uninstaller
Thanks for taking care of that so quickly, Isaiah! -E On Fri, Aug 22, 2014 at 4:12 AM, Isaiah Norton isaiah.nor...@gmail.com wrote: The (un)installer system has been updated and the download links have been restored. On Thu, Aug 21, 2014 at 7:58 PM, Isaiah Norton isaiah.nor...@gmail.com wrote: Please be aware that the Windows uninstaller for the 0.3-dev series and the initial 0.3 release has a dangerous bug that could cause loss of non-julia files in the install directory. * DO NOT run the uninstaller if there are non-Julia files in the install directory. * To be safe, it is recommended to remove the uninstaller.exe file. If you wish to uninstall in the future, manually removing the install directory (if you deem safe) and the Start Menu entry will accomplish the same thing. Explanation: the uninstaller will remove *all* files and directories within the install directory. For example, if the install directory is changed from the default to an existing directory, any files or subdirectories within that existing directory will be removed when the uninstaller is executed. As a precaution, we have disabled the 0.3 download link. We have a fix in progress and the download should be available again within a few hours. Isaiah
[julia-users] Conditional import within a function
I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] why sum(abs(A)) is very slow
On Fri, Aug 22, 2014 at 11:32 AM, Rafael Fourquet fourquet.raf...@gmail.com wrote: My naive answer is then why not make vectorized functions lazy (like iabs above, plus dimensions information) by default? Do you have links to relevant discussions? If that was the way things worked, would sum(abs(A)) do the computation right away or just wait until you ask for the result? In other words, should sum also be lazy if we're doing all vectorized computations that way? What about sum(abs(A),1)? Lazy or eager? What about A*B when A and B are matrices? Should that be an eager matrix product or just a lazy representation that hangs onto A and B and answers queries about their product on demand? If you're computing trace(A*B) then you can save a huge amount of work that way. But if you need all or most of the values in A*B then computing each one as a vector-vector product on demand is very inefficient.
[julia-users] Re: Conditional import within a function
From https://groups.google.com/d/topic/julia-users/AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] Conditional import within a function
What I've been doing is very similar. I leave the plotting package up to the user, but they are first required to do one of: using PyPlot or using Gadfly inside their script. I then have a function similar to yours (named plot(x::MyType), to take advantage of multiple dispatch) that checks for isdefined(:PyPlot) or isdefined(:Gadfly). If neither is defined I throw an error telling the user to include a plotting package in their script. This keeps load times quick if they are not using plotting functionality. To answer you question, conditional import inside functions is not possible. On Fri, Aug 22, 2014 at 11:56 AM, Spencer Lyon spencerly...@gmail.com wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
[julia-users] Re: Conditional import within a function
Thanks, good workaround. On Friday, August 22, 2014 12:16:03 PM UTC-4, Peter Simon wrote: From https://groups.google.com/d/topic/julia-users/AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] Re: Conditional import within a function
Peter Simon, very cool. When I ran hit this problem I saw it as an opportunity to make my code more Julian. :P Eval FTW. On Fri, Aug 22, 2014 at 12:16 PM, Peter Simon psimon0...@gmail.com wrote: From https://groups.google.com/d/topic/julia-users/AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] Conditional import within a function
Thanks Steve, That is a good solution. I think I will probably end up doing this, or maybe even a combination of what you and Peter said and instead of throwing an error if it isn't defined, I will just define it for them. On Friday, August 22, 2014 12:16:14 PM UTC-4, Steve Kelly wrote: What I've been doing is very similar. I leave the plotting package up to the user, but they are first required to do one of: using PyPlot or using Gadfly inside their script. I then have a function similar to yours (named plot(x::MyType), to take advantage of multiple dispatch) that checks for isdefined(:PyPlot) or isdefined(:Gadfly). If neither is defined I throw an error telling the user to include a plotting package in their script. This keeps load times quick if they are not using plotting functionality. To answer you question, conditional import inside functions is not possible. On Fri, Aug 22, 2014 at 11:56 AM, Spencer Lyon spence...@gmail.com javascript: wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
[julia-users] Re: Call for participation: High Performance Technical Computing in Dynamic Languages workshop (11/17/2014)
I'm not an organizer of this workshop, but just wanted to remind everyone of this and note that the deadline for submission is now August 25th, so get to writing those papers! On Monday, March 31, 2014 10:14:21 AM UTC-6, Jiahao Chen wrote: It is my great pleasure to announce a workshop at this year's Supercomputing 2014 conference entitled High Performance Technical Computing in Dynamic Languages which will be held on Monday, November 17, 2014 at the Colorado Convention Center in New Orleans, Louisiana. On behalf of the organizing committee, I hope you will be able to attend the workshop to meet other Julia developers and enthusiasts, and to find common ground with practitioners of other related languages such as Python and R. We are also soliciting papers regardless of discipline, affiliation or language (within the scope of high level dynamic languages) pertaining to the scalability challenges of deploying code written in high level dynamic languages. Please refer to the conference website for further details: http://jiahao.github.io/hptcdl-sc14 Thanks, Jiahao Chen Staff Research Scientist MIT Computer Science and Artificial Intelligence Laboratory
Re: [julia-users] Re: Conditional import within a function
I’m actually still having issues with both of these options — I’ll try to enumerate what I think the problem is here. - When I do using MyPackage and my code is loaded (including the plotting routines). - Then when I call plot(x:MyType) (sorry for the shorthand), the function is compiled. - If PyPlot is not defined I try to either define it for them or emit an error message telling them to define it. - After PyPlot is defined I try to run the file again, and I get errors telling me the functions from PyPlot that I use in the plot(x::MyType) function are not defined. I think the reason they are not defined within the function is that it was compiled the first time I called it, when the functions weren’t actually available. Does my analysis seem correct? Does anyone know a way to accomplish this? -- I will provide a quick usable example so it is easy for people to experiment type Foo x::Int end function check_defined_err(x::Symbol) if !isdefined(x) error(Module $x not defined. run `using $x` to fix the problem) end end function check_defined_eval(x::Symbol) if !isdefined(x) eval(Expr(:using, x)) end end function plot_err(x::Foo) check_defined_err(:PyPlot) d = 1:x plot(d, d.^2) end function plot_eval(x::Foo) check_defined_eval(:PyPlot) d = 1:x plot(d, d.^2) end a = Foo(10) # plot_err(a) # plot_eval(a) Then when I have run the code above, I get the following at the console (it says string because I : julia plot_err(a) ERROR: Module PyPlot not defined. run `using PyPlot` to fix the problem in error at /usr/local/julia/usr/lib/julia/sys.dylib in check_defined_err at none:7 in plot_err at string:18 julia using PyPlot INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. julia plot_err(a) ERROR: plot not defined in plot_err at none:20 In another session to see plot_eval(a): julia plot_eval(a) INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. ERROR: plot not defined in plot_eval at none:4 Thank you for your help. On Friday, August 22, 2014 12:18:14 PM UTC-4, Steve Kelly wrote: Peter Simon, very cool. When I ran hit this problem I saw it as an opportunity to make my code more Julian. :P Eval FTW. On Fri, Aug 22, 2014 at 12:16 PM, Peter Simon psimo...@gmail.com javascript: wrote: From https://groups.google.com/d/topic/julia-users/AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] why sum(abs(A)) is very slow
I'm not familiar with lazy evaluation (I've not used any language implementing it). But I was wondering... Why not have a 'calculate_now' function to let the programmer choose when a result is guaranteed to be calculated? Otherwise, resort to lazy representations. There could be some heuristic also: if at least one of the original object is freed by the GC, perform all the calculations depending on it. That could also be simpler: defer actual calculations until the end of the current block.
Re: [julia-users] Manipulations with abstract parametric types
Instead of eltype as written, use the following format:
eltype{T:...}(::Type{T}) = eltype(super(T))
This should help with the type inference issue too.
I think M.name is already the type, so you don't need to use eval (which
doesn't return the right thing anyways, in the general case)
Sorry I can't type more clearly on a phone, but I'm hoping this will be
enough to get you going in the right direction.
On Friday, August 22, 2014, Tim Holy tim.h...@gmail.com wrote:
Hi all,
I've come up against a couple of surprising type-manipulation issues and
I'm
wondering if I'm just missing something. Hoping someone out there knows a
better way of doing these manipulations.
The issues arise when I try writing generic algorithms on abstract
parametric
types that need to make decisions based on both the concrete parameters and
the concrete type. Let's start with a simple example:
abstract MyAbstractType{T}
immutable MyType{T} : MyAbstractType{T}
a::T
b::T
end
Now let's write a generic eltype method:
Base.eltype{M:MyAbstractType}(::Type{M}) = M.parameters[1]
It's a little ugly to have to use M.parameters[1] here. While I'm curious
to
know whether I'm missing a simple alternative, I can live with this.
Now let's define some arithmetic operations:
(*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
(.*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
And then try it out:
julia y = MyType(3,12)
julia 2y
MyType{Int64}(6,24)
julia 2.1y
MyType{Float64}(6.301,25.203)
So far, so good. Now let's try something a little more sophisticated:
julia A = [MyType(3,12), MyType(4,7)]
2-element Array{MyType{Int64},1}:
MyType{Int64}(3,12)
MyType{Int64}(4,7)
julia 2A
2-element Array{Any,1}:
MyType{Int64}(6,24)
MyType{Int64}(8,14)
The problem here is we're getting an Array{Any,1} back. No problem you say,
let's try to help type inference out, by specifying that a
::MyType{Int}*::Float64 is a MyType{Float64}. If we were willing to write
this
using concrete types, it would be easy:
Base.promote_array_type{T:Real, S}(::Type{T}, ::Type{MyType{S}}) =
MyType{promote_type(T, S)}
But if we want to just use the abstract type, then I've been unsuccessful
in
avoiding this construction:
Base.promote_array_type{T:Real, M:MyAbstractType}(::Type{T}, ::Type{M}) =
eval(M.name.name){promote_type(eltype(M), T)}
Ouch! Do we really have to use eval there? This seems likely to be a
runtime
bottleneck, which seems confirmed by
code_llvm(Base.promote_array_type, (Type{Float64}, Type{eltype(A)}))
Compare it against the version generated when I instead use the concrete
type,
and you'll see what I mean.
Of course these issues would be easily solved by triangular dispatch or
staged
functions, but I'm hoping there's another good way that exists today.
--Tim
Re: [julia-users] Re: Conditional import within a function
Changing plot() to PyPlot.plot() fixes it for me. type Foo x::Int end function check_defined_err(x::Symbol) if !isdefined(x) error(Module $x not defined. run `using $x` to fix the problem) end end function check_defined_eval(x::Symbol) if !isdefined(x) eval(Expr(:using, x)) end end function plot_err(x::Foo) check_defined_err(:PyPlot) d = 1:x.x PyPlot.plot(d, d.^2) end function plot_eval(x::Foo) check_defined_eval(:PyPlot) d = 1:x.x PyPlot.plot(d, d.^2) end a = Foo(10) try plot_err(a) catch err println(err) end plot_eval(a) This also eliminates the warning: Warning: using PyPlot.plot in module Main conflicts with an existing identifier. Which makes me think you analysis is correct. When Julia evaluates the function it must be binding the name plot to something. Maybe somebody more knowledgeable about the pipeline can chime in. I don't think this is a scoping issue, as much as it is intrinsic to the eval and compilation steps. On Fri, Aug 22, 2014 at 12:49 PM, Spencer Lyon spencerly...@gmail.com wrote: I’m actually still having issues with both of these options — I’ll try to enumerate what I think the problem is here. - When I do using MyPackage and my code is loaded (including the plotting routines). - Then when I call plot(x:MyType) (sorry for the shorthand), the function is compiled. - If PyPlot is not defined I try to either define it for them or emit an error message telling them to define it. - After PyPlot is defined I try to run the file again, and I get errors telling me the functions from PyPlot that I use in the plot(x::MyType) function are not defined. I think the reason they are not defined within the function is that it was compiled the first time I called it, when the functions weren’t actually available. Does my analysis seem correct? Does anyone know a way to accomplish this? -- I will provide a quick usable example so it is easy for people to experiment type Foo x::Int end function check_defined_err(x::Symbol) if !isdefined(x) error(Module $x not defined. run `using $x` to fix the problem) end end function check_defined_eval(x::Symbol) if !isdefined(x) eval(Expr(:using, x)) end end function plot_err(x::Foo) check_defined_err(:PyPlot) d = 1:x plot(d, d.^2) end function plot_eval(x::Foo) check_defined_eval(:PyPlot) d = 1:x plot(d, d.^2) end a = Foo(10) # plot_err(a) # plot_eval(a) Then when I have run the code above, I get the following at the console (it says string because I : julia plot_err(a) ERROR: Module PyPlot not defined. run `using PyPlot` to fix the problem in error at /usr/local/julia/usr/lib/julia/sys.dylib in check_defined_err at none:7 in plot_err at string:18 julia using PyPlot INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. julia plot_err(a) ERROR: plot not defined in plot_err at none:20 In another session to see plot_eval(a): julia plot_eval(a) INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. ERROR: plot not defined in plot_eval at none:4 Thank you for your help. On Friday, August 22, 2014 12:18:14 PM UTC-4, Steve Kelly wrote: Peter Simon, very cool. When I ran hit this problem I saw it as an opportunity to make my code more Julian. :P Eval FTW. On Fri, Aug 22, 2014 at 12:16 PM, Peter Simon psimo...@gmail.com wrote: From https://groups.google.com/d/topic/julia-users/ AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] Re: Conditional import within a function
Ok thank you for working on it. I will just use PyPlot.plot for now and see if anyone can help us understand what is happening here. On Friday, August 22, 2014 1:24:35 PM UTC-4, Steve Kelly wrote: Changing plot() to PyPlot.plot() fixes it for me. type Foo x::Int end function check_defined_err(x::Symbol) if !isdefined(x) error(Module $x not defined. run `using $x` to fix the problem) end end function check_defined_eval(x::Symbol) if !isdefined(x) eval(Expr(:using, x)) end end function plot_err(x::Foo) check_defined_err(:PyPlot) d = 1:x.x PyPlot.plot(d, d.^2) end function plot_eval(x::Foo) check_defined_eval(:PyPlot) d = 1:x.x PyPlot.plot(d, d.^2) end a = Foo(10) try plot_err(a) catch err println(err) end plot_eval(a) This also eliminates the warning: Warning: using PyPlot.plot in module Main conflicts with an existing identifier. Which makes me think you analysis is correct. When Julia evaluates the function it must be binding the name plot to something. Maybe somebody more knowledgeable about the pipeline can chime in. I don't think this is a scoping issue, as much as it is intrinsic to the eval and compilation steps. On Fri, Aug 22, 2014 at 12:49 PM, Spencer Lyon spence...@gmail.com javascript: wrote: I’m actually still having issues with both of these options — I’ll try to enumerate what I think the problem is here. - When I do using MyPackage and my code is loaded (including the plotting routines). - Then when I call plot(x:MyType) (sorry for the shorthand), the function is compiled. - If PyPlot is not defined I try to either define it for them or emit an error message telling them to define it. - After PyPlot is defined I try to run the file again, and I get errors telling me the functions from PyPlot that I use in the plot(x::MyType) function are not defined. I think the reason they are not defined within the function is that it was compiled the first time I called it, when the functions weren’t actually available. Does my analysis seem correct? Does anyone know a way to accomplish this? -- I will provide a quick usable example so it is easy for people to experiment type Foo x::Int end function check_defined_err(x::Symbol) if !isdefined(x) error(Module $x not defined. run `using $x` to fix the problem) end end function check_defined_eval(x::Symbol) if !isdefined(x) eval(Expr(:using, x)) end end function plot_err(x::Foo) check_defined_err(:PyPlot) d = 1:x plot(d, d.^2) end function plot_eval(x::Foo) check_defined_eval(:PyPlot) d = 1:x plot(d, d.^2) end a = Foo(10) # plot_err(a) # plot_eval(a) Then when I have run the code above, I get the following at the console (it says string because I : julia plot_err(a) ERROR: Module PyPlot not defined. run `using PyPlot` to fix the problem in error at /usr/local/julia/usr/lib/julia/sys.dylib in check_defined_err at none:7 in plot_err at string:18 julia using PyPlot INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. julia plot_err(a) ERROR: plot not defined in plot_err at none:20 In another session to see plot_eval(a): julia plot_eval(a) INFO: Loading help data... Warning: using PyPlot.plot in module Main conflicts with an existing identifier. ERROR: plot not defined in plot_eval at none:4 Thank you for your help. On Friday, August 22, 2014 12:18:14 PM UTC-4, Steve Kelly wrote: Peter Simon, very cool. When I ran hit this problem I saw it as an opportunity to make my code more Julian. :P Eval FTW. On Fri, Aug 22, 2014 at 12:16 PM, Peter Simon psimo...@gmail.com wrote: From https://groups.google.com/d/topic/julia-users/ AWCerAdDLQo/discussion : eval(Expr(:using,:PyPlot)) can be used inside a conditional. --Peter On Friday, August 22, 2014 8:56:53 AM UTC-7, Spencer Lyon wrote: I am working on a library that defines various types as well as a few “helper” functions to plot those types with PyPlot. If I do [import|using] PyPlot at the top level of any file in my package, PyPlot is loaded when I do [using|import] MyPackage. This makes the startup time for my package much much longer. What I would like to do is instead of having to load it when my package loads, I could load it when someone calls one of the functions that needs it. Here is an example of what I would like to do: function plot_my_type(x::MyType) if !isdefined(:PyPlot) using PyPlot end # finish the function by plotting with PyPlot end I haven’t been able to get a solution that works for this. Does anyone know if it is possible?
Re: [julia-users] why sum(abs(A)) is very slow
If that was the way things worked, would sum(abs(A)) do the computation right away or just wait until you ask for the result? In other words, should sum also be lazy if we're doing all vectorized computations that way? sum(abs(A)) returns a scalar, so lazy would buy nothing here (in most cases at least, let's not be haskell!) What about sum(abs(A),1)? Lazy or eager? If dim A1, the result is an array so lazy. In short, be lazy when it gives opportunity for loop fusion, and saves allocations. What about A*B when A and B are matrices? I was more thinking of operations done element-wise (of the form of map(f, A1, ...), like abs and +). Optimizing a product A*B is less trivial (C++ expressions templates...), si I prefer not answer!
Re: [julia-users] Re: function version of
How is parsed has nothing to do with whether we use a parser generator or not. The only relevance is that having a formal grammar would make it easier to know that is special in this respect. But better documentation would serve that purpose better than a formal grammar. On Aug 21, 2014, at 1:05 PM, gentlebeldin gentlebel...@hotmail.com wrote: Whatever, if I rtfm, especially the part saying Operators Are Functions (with the only exceptions of and ||, because of short-circuit evaluation), then I conclude that the error message following (x,y) is not a feature, it's a bug. And whether we like a generated parser or not, it's better than a buggy one, imho. Sure grammar can be ambiguous, but there are techniques to get around that, like here: http://accent.compilertools.net/Accent.html
Re: [julia-users] why sum(abs(A)) is very slow
Could you please explain why the iterator version is so much faster? Is it simply from avoiding temporary array allocation? That's what I understand, and maybe marginally because there is only one pass over the data.
Re: [julia-users] Manipulations with abstract parametric types
On Friday, August 22, 2014 01:17:53 PM Jameson Nash wrote:
Instead of eltype as written, use the following format:
eltype{T:...}(::Type{T}) = eltype(super(T))
With this definition:
Base.eltype{T:MyAbstractType}(::Type{T}) = eltype(super(T))
eltype(MyType{Float32}) yields Any.
And for the second:
julia f{M:MyAbstractType}(::Type{M}) = (M.name){Float32}
f (generic function with 1 method)
julia f(MyType{Float64})
ERROR: type: instantiate_type: expected TypeConstructor, got TypeName
in f at none:1
It was a good suggestion, it's just that I had already tried it.
--Tim
This should help with the type inference issue too.
I think M.name is already the type, so you don't need to use eval (which
doesn't return the right thing anyways, in the general case)
Sorry I can't type more clearly on a phone, but I'm hoping this will be
enough to get you going in the right direction.
On Friday, August 22, 2014, Tim Holy tim.h...@gmail.com wrote:
Hi all,
I've come up against a couple of surprising type-manipulation issues and
I'm
wondering if I'm just missing something. Hoping someone out there knows a
better way of doing these manipulations.
The issues arise when I try writing generic algorithms on abstract
parametric
types that need to make decisions based on both the concrete parameters
and
the concrete type. Let's start with a simple example:
abstract MyAbstractType{T}
immutable MyType{T} : MyAbstractType{T}
a::T
b::T
end
Now let's write a generic eltype method:
Base.eltype{M:MyAbstractType}(::Type{M}) = M.parameters[1]
It's a little ugly to have to use M.parameters[1] here. While I'm curious
to
know whether I'm missing a simple alternative, I can live with this.
Now let's define some arithmetic operations:
(*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
(.*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
And then try it out:
julia y = MyType(3,12)
julia 2y
MyType{Int64}(6,24)
julia 2.1y
MyType{Float64}(6.301,25.203)
So far, so good. Now let's try something a little more sophisticated:
julia A = [MyType(3,12), MyType(4,7)]
2-element Array{MyType{Int64},1}:
MyType{Int64}(3,12)
MyType{Int64}(4,7)
julia 2A
2-element Array{Any,1}:
MyType{Int64}(6,24)
MyType{Int64}(8,14)
The problem here is we're getting an Array{Any,1} back. No problem you
say,
let's try to help type inference out, by specifying that a
::MyType{Int}*::Float64 is a MyType{Float64}. If we were willing to write
this
using concrete types, it would be easy:
Base.promote_array_type{T:Real, S}(::Type{T}, ::Type{MyType{S}}) =
MyType{promote_type(T, S)}
But if we want to just use the abstract type, then I've been unsuccessful
in
avoiding this construction:
Base.promote_array_type{T:Real, M:MyAbstractType}(::Type{T}, ::Type{M})
=
eval(M.name.name){promote_type(eltype(M), T)}
Ouch! Do we really have to use eval there? This seems likely to be a
runtime
bottleneck, which seems confirmed by
code_llvm(Base.promote_array_type, (Type{Float64}, Type{eltype(A)}))
Compare it against the version generated when I instead use the concrete
type,
and you'll see what I mean.
Of course these issues would be easily solved by triangular dispatch or
staged
functions, but I'm hoping there's another good way that exists today.
--Tim
[julia-users] Re: Semicolon prefix for shell command no longer working in IJulia Notebook
Solved it. Shell mode works in IJulia notebook, but only for a single command in each cell.
Re: [julia-users] why sum(abs(A)) is very slow
On Aug 22, 2014, at 1:45 PM, Rafael Fourquet fourquet.raf...@gmail.com wrote: In short, be lazy when it gives opportunity for loop fusion, and saves allocations. There's a complicated limit to when you want to fuse loops – at some point multiple iterations becomes better than fused loops and it all depends on how much and what kind of work you're doing. In general doing things lazily does not cut down on allocation since you have to allocate the representation of the operations that you're deferring and close over any values that they depend on. This particular example only works out so well because the iterable is so simple that the compiler can eliminate the laziness and do the eager loop fused version for you. This will not generally be the case. You're welcome to experiment (and Julia's type system makes it pretty easy to do so), but I think that you'll quickly find that more laziness is not a panacea for performance problems.
[julia-users] Re: reference for the Formula language used in MixedModels
On Thursday, August 21, 2014 4:41:02 PM UTC-5, Thomas Covert wrote:
Thanks for the thorough explanation. To be clear, though, if f is a
PooledDataFactor, f is treated as a fixed effect in the Formula language,
whereas (1|f) is treated as a random effect?
That's correct, except that the name of the type is PooledDataArray.
I should have been more specific about what are fixed-effects terms and
what are random-effects terms. A random-effects term is distinguished by
the vertical bar, |. The precedence of operators requires that the
random-effects expression be enclosed in parentheses. The expression to
the left of the vertical bar is evaluated as a model matrix according to
the rules of the R formula language. In particular, the intercept term,
which generates a column of 1's, is implicit. Hence (1+x|g) and (x|g) are
equivalent and generate random slopes w.r.t. x and random intercepts for
each level of g. (Perhaps we should change this but that is the way it is
now). To suppress an intercept term you write (0+x|g).
A fixed-effects term, f, where f is a PooledDataVector with k levels,
generates k-1 contrast columns. (If you know the technical definition of
a contrast as used in old analysis of variance descriptions these are not
really contrasts in that sense but that is irrelevant here.) Again, this
is because of the implied intercept 1 + f is equivalent to f, and 1+f
generates k columns consisting of the intercept column and k-1 of the k
indicator columns for the levels of f. We drop the first column, again
following the R convention where the so-called treatment contrasts are
the default. R allows other contrast specifications. We haven't yet added
that capability to the formula language in Julia.
It happens that 0+f also generates k columns, which are the full set of
indicator columns.
Similarly, fg is the cartesian product of the fixed effects for f and
g?
A interaction term like fg is rarely used by itself. The more common form
is f*g which expands to the main effects and the second order interaction.
That is f*g expands to 1 + f + g + fg. In this case fg is the Cartesian
product of the contrasts columns. This corresponds to a two-factor
analysis of variance with interaction. If g has l levels then the main
effects for f have k-1 degrees of freedom, the main effects for g have l-1
degrees of freedom and the interaction term has (k-1)(l-1) degrees of
freedom (assuming you have at least k*l observations and the model matrix
is of full rank). In these cases degrees of freedom means what it should
mean, the dimension of a linear subspace of the sample space.
I see that (x|g) has random effects on the slope of x. Is there a way to
get fixed slopes on x?
The full model expression would be
y ~ 1 + x + (1+x|g)
generating two fixed effects, the population-wide intercept and slope
w.r.t. x and two random effects (change in the intercept and slope) for
each level of g. See the example of the sleepstudy data in the README.md
for the MixedModels package.
Let's take any further discussion of the model formula language to the
Julia-Stats group so as not to frighten those who remember with horror
their introductory statistics course.
On Thursday, August 21, 2014 4:35:05 PM UTC-5, Douglas Bates wrote:
On Thursday, August 21, 2014 4:20:46 PM UTC-5, Thomas Covert wrote:
Is there a reference somewhere for the formula language specified in
DataFrames and used in MixedModels? In particular, I'm confused about how
fixed- and random-effects are separately specified. For example, suppose
I've got Y_{it} = X_{it}b + u_i + e_{it}. My understanding is that a fixed
effect spec for this is Y ~ X + (1|i). What is the random effects
specification? If I had a separate categorical variable Z, how would I
write Y_{it} = X_{it}b + {Fixed effects on categories of Z} + u_i + e_{it},
with random effects on i?
The formula language used in MixedModels is similar to that used in the
lme4 package for R. There are examples in the README.md file and in the
demo and docs directories of the package.
To specify random effects you need to have a factor (PooledDataVector in
Julia parlance) which would correspond to the i subscript in your
specification. Call this g and the response y. Then a simple random
effects model is written as
y ~ 1 + (1|g)
A model with covariates, x (numeric) and f (categorical) is wiitten
y ~ x + f + (1|g)
A model with these covariates, random slopes and random intercepts is
written
y ~ x + f + (x|g)
Perhaps it would be best to use the issue tracker in the MixedModels
package if you have further questions of this type.
Thanks.
-Thom
[julia-users] Re: Installing Julia on Ubuntu 13.04 through apt-get
Thanks it worked! I now have upgraded linux to 14.04. and run Julia version 0.3.0 However for whom that may interest apt-cache show julia- still reference: Version: 0.2.1+dfsg-2 Thanks again. -Eric On Wednesday, August 20, 2014 12:08:17 PM UTC-7, Patrick O'Leary wrote: Raring went EOL in January; if you're stuck on it, you might be able to switch over to a different package repository URL (see http://askubuntu.com/questions/499712/is-distribution-of-raring-packages-officially-ended), but you should seriously consider upgrading instead. Note that the latest Ubuntu release, 14.04 Trusty Tahr, is a LTS release and will be supported until April 2019--I would recommend using this version. On Wednesday, August 20, 2014 1:21:54 PM UTC-5, eric l wrote: After running Julia on MacOS decided to give a try on my Linux machine (i7 Haswell running 13.04 raring). sudo apt-get install julia I got the following: Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libamd2.2.0 libarpack2 libcholmod1.7.1 libcolamd2.7.1 libdouble-conversion0 libfftw3-3 liblapack3 libopenblas-base librmath-base librmath-dev libumfpack5.4.0 libunwind8 Suggested packages: ess libfftw3-bin libfftw3-dev The following NEW packages will be installed: julia libamd2.2.0 libarpack2 libcholmod1.7.1 libcolamd2.7.1 libdouble-conversion0 libfftw3-3 liblapack3 libopenblas-base librmath-base librmath-dev libumfpack5.4.0 libunwind8 0 upgraded, 13 newly installed, 0 to remove and 0 not upgraded. Need to get 5,252 kB/22.3 MB of archives. After this operation, 72.1 MB of additional disk space will be used. Do you want to continue [Y/n]? y WARNING: The following packages cannot be authenticated! libunwind8 libdouble-conversion0 liblapack3 libfftw3-3 libarpack2 Install these packages without verification [y/N]? y Err http://us.archive.ubuntu.com/ubuntu/ raring/main libunwind8 amd64 1.0.1-4ubuntu2 404 Not Found [IP: 91.189.91.13 80] Err http://us.archive.ubuntu.com/ubuntu/ raring/main liblapack3 amd64 3.4.2-1~exp3 404 Not Found [IP: 91.189.91.13 80] Err http://us.archive.ubuntu.com/ubuntu/ raring/universe libfftw3-3 amd64 3.3.3-2ubuntu1 404 Not Found [IP: 91.189.91.13 80] Err http://us.archive.ubuntu.com/ubuntu/ raring/universe libarpack2 amd64 3.1.2-2~exp1 404 Not Found [IP: 91.189.91.13 80] Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/libu/libunwind/libunwind8_1.0.1-4ubuntu2_amd64.deb 404 Not Found [IP: 91.189.91.13 80] Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/l/lapack/liblapack3_3.4.2-1~exp3_amd64.deb 404 Not Found [IP: 91.189.91.13 80] Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/universe/f/fftw3/libfftw3-3_3.3.3-2ubuntu1_amd64.deb 404 Not Found [IP: 91.189.91.13 80] Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/universe/a/arpack/libarpack2_3.1.2-2~exp1_amd64.deb 404 Not Found [IP: 91.189.91.13 80] Unable to correct missing packages. E: Aborting install. So I check the us.archive.ubuntu.com and indeed path and/or file just do not exist. For example their is no raring directory under us.archive.ubuntu.com/ubuntu/ I get the same kind of error if I run a apt-get update. What am I missing? Please advise. Thanks, -Eric
[julia-users] RFC: Colored output API
I just filed an issue [1] that currently the print_with_color() doesn't seem to use the correct colors on my system. While checking it out, I was thinking that print_with_color might fit in nicely with Color.jl. We could pull print_with_color out of Base and add all the color terminal codes to Color.jl, which could implement print(msg::String, color::ColorValue) and related functions. This feels more in line with Julian style of leveraging the type system to replace functions with underscores. [1]: https://github.com/JuliaLang/julia/issues/8096
Re: [julia-users] RFC: Colored output API
Abstractly, I’d love to be able to print using Color.jl. It seems like a much cleaner interface. But it’s not clear to me that Color.jl buys you a lot if your terminal doesn’t support a widerange of colors. — John On Aug 22, 2014, at 3:33 PM, Spencer Russell spencer.f.russ...@gmail.com wrote: I just filed an issue [1] that currently the print_with_color() doesn't seem to use the correct colors on my system. While checking it out, I was thinking that print_with_color might fit in nicely with Color.jl. We could pull print_with_color out of Base and add all the color terminal codes to Color.jl, which could implement print(msg::String, color::ColorValue) and related functions. This feels more in line with Julian style of leveraging the type system to replace functions with underscores. [1]: https://github.com/JuliaLang/julia/issues/8096
[julia-users] suggestion for good algorithms?
Er, there is an almost-close-form solution no? Start with (n(odd))C(n/2+-2.5) Tony
Re: [julia-users] suggestion for good algorithms?
Thanks for the response, but sorry, can you clarify what that formula means? On 2014年08月23日 09:24, Tony Fong wrote: Er, there is an almost-close-form solution no? Start with (n(odd))C(n/2+-2.5) Tony
[julia-users] Re: suggestion for good algorithms?
(If this is for a real application and not just an exercise, you probably want to just do something directly with the `nonzeros` array inside the loop rather than pushing copies to a long array, especially if N and K are larger.)
[julia-users] Best way to convert Any of vectors into matrix
Good evening,
I often have Any types filled with vectors (especially as the return of a
pmap), and need them as a matrix or DataFrame. For example, suppose I have,
julia A
3-element Array{Any,1}:
[1,2]
[3,4]
[5,6]
But I want,
julia B
2x3 Array{Int64,2}:
1 3 5
2 4 6
I came up with the following, which successfully constructs B from A, but
it's a bit inefficient/messy:
B = A[1]
for i=2:length(A)
B = hcat(B,A[i])
end
Is there a better way to do this? Something like,
julia B = hcatForAny(A)
Thanks,
Bradley
[julia-users] Re: Best way to convert Any of vectors into matrix
A = Any[[1,2],[3,4],[5,6]]
function to_matrix(A::Vector{Any})
ncol = length(A)
nrow = length(A[1])
B = Array( typeof(A[1][1]) , (nrow, ncol) )
for i in 1:ncol
for j in 1:nrow
B[j,i] = A[i][j]
end
end
B
end
to_matrix(A)
2x3 Array{Int64,2}:
1 3 5
2 4 6
function use_hcat(A::Vector{Any})
B = A[1]
for i=2:length(A)
B = hcat(B,A[i])
end
B
end
use_hcat(A)
2x3 Array{Int64,2}:
1 3 5
2 4 6
@time to_matrix(A)
elapsed time: 1.1926e-5 seconds (504 bytes allocated)
@time use_hcat(A)
elapsed time: 8.743e-6 seconds (288 bytes allocated)
El viernes, 22 de agosto de 2014 23:05:19 UTC-3, Bradley Setzler escribió:
Good evening,
I often have Any types filled with vectors (especially as the return of a
pmap), and need them as a matrix or DataFrame. For example, suppose I have,
julia A
3-element Array{Any,1}:
[1,2]
[3,4]
[5,6]
But I want,
julia B
2x3 Array{Int64,2}:
1 3 5
2 4 6
I came up with the following, which successfully constructs B from A, but
it's a bit inefficient/messy:
B = A[1]
for i=2:length(A)
B = hcat(B,A[i])
end
Is there a better way to do this? Something like,
julia B = hcatForAny(A)
Thanks,
Bradley
Re: [julia-users] Manipulations with abstract parametric types
Upon reflection, I realized that your second one wasn't working because it
is badly posed. Your definition implicitly assumes a lot about the
structure and fields of any subtype that may not be true, or may have
multiple answers.
For example, with the following abstract definition:
abstract MyAbstract{T}
I can define all of the following:
type MyType1{T} : MyAbstract{T} end
type MyType2 : MyAbstract{Int} end
type MyType3{T} : MyAbstract{Float32} end
type MyType4{T,S} : MyAbstract{S} end
In short, the type parameter of a subtype has little or nothing to do with
the type parameter (or eltype) of the abstract type. Therefore, it is not a
meaningful statement to write a generic promote_type function in the way
that you were proposing.
Your original eltype function would have returned the incorrect eltype for
all of these, but the eltype based upon super, is able to use dispatch to
return the correct eltype, regardless of how the subtypes are defined.
On Fri, Aug 22, 2014 at 3:45 PM, Tim Holy tim.h...@gmail.com wrote:
Got it now, that's a good strategy.
Of course, it's the second one I'm more concerned about.
--Tim
On Friday, August 22, 2014 03:18:19 PM Jameson Nash wrote:
For the first, you are missing the base case for the recursion, so it
defaults to eltype(::Any) instead of eltype(::Type{MyAbstract{T}})
On Friday, August 22, 2014, Tim Holy tim.h...@gmail.com wrote:
On Friday, August 22, 2014 01:17:53 PM Jameson Nash wrote:
Instead of eltype as written, use the following format:
eltype{T:...}(::Type{T}) = eltype(super(T))
With this definition:
Base.eltype{T:MyAbstractType}(::Type{T}) = eltype(super(T))
eltype(MyType{Float32}) yields Any.
And for the second:
julia f{M:MyAbstractType}(::Type{M}) = (M.name){Float32}
f (generic function with 1 method)
julia f(MyType{Float64})
ERROR: type: instantiate_type: expected TypeConstructor, got TypeName
in f at none:1
It was a good suggestion, it's just that I had already tried it.
--Tim
This should help with the type inference issue too.
I think M.name is already the type, so you don't need to use eval
(which
doesn't return the right thing anyways, in the general case)
Sorry I can't type more clearly on a phone, but I'm hoping this will
be
enough to get you going in the right direction.
On Friday, August 22, 2014, Tim Holy tim.h...@gmail.com
javascript:;
wrote:
Hi all,
I've come up against a couple of surprising type-manipulation
issues
and
I'm
wondering if I'm just missing something. Hoping someone out there
knows a
better way of doing these manipulations.
The issues arise when I try writing generic algorithms on abstract
parametric
types that need to make decisions based on both the concrete
parameters
and
the concrete type. Let's start with a simple example:
abstract MyAbstractType{T}
immutable MyType{T} : MyAbstractType{T}
a::T
b::T
end
Now let's write a generic eltype method:
Base.eltype{M:MyAbstractType}(::Type{M}) = M.parameters[1]
It's a little ugly to have to use M.parameters[1] here. While I'm
curious
to
know whether I'm missing a simple alternative, I can live with
this.
Now let's define some arithmetic operations:
(*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
(.*)(x::Number, m::MyType) = MyType(x*m.a, x*m.b)
And then try it out:
julia y = MyType(3,12)
julia 2y
MyType{Int64}(6,24)
julia 2.1y
MyType{Float64}(6.301,25.203)
So far, so good. Now let's try something a little more
sophisticated:
julia A = [MyType(3,12), MyType(4,7)]
2-element Array{MyType{Int64},1}:
MyType{Int64}(3,12)
MyType{Int64}(4,7)
julia 2A
2-element Array{Any,1}:
MyType{Int64}(6,24)
MyType{Int64}(8,14)
The problem here is we're getting an Array{Any,1} back. No problem
you
say,
let's try to help type inference out, by specifying that a
::MyType{Int}*::Float64 is a MyType{Float64}. If we were willing to
write
this
using concrete types, it would be easy:
Base.promote_array_type{T:Real, S}(::Type{T}, ::Type{MyType{S}}) =
MyType{promote_type(T, S)}
But if we want to just use the abstract type, then I've been
unsuccessful
in
avoiding this construction:
Base.promote_array_type{T:Real, M:MyAbstractType}(::Type{T},
::
::Type{M})
::
=
eval(M.name.name){promote_type(eltype(M), T)}
Ouch! Do we really have to use eval there? This seems likely to be
a
runtime
bottleneck, which seems confirmed by
code_llvm(Base.promote_array_type, (Type{Float64},
[julia-users] possible timing regression in A\b
Is anyone else seeing the following? If not, what could I have done to my env to trigger it? Thx. julia VERSION v0.3.0 julia @time begin a = rand(5000,5000); b = rand(5000); x =a\b end; elapsed time: 31.413347385 seconds (440084348 bytes allocated, 0.12% gc time) julia julia VERSION v0.4.0-dev+308 julia @time begin a = rand(5000,5000); b = rand(5000); x =a\b end; elapsed time: 1.686715561 seconds (431769824 bytes allocated, 0.87% gc time) julia
Re: [julia-users] RFC: Colored output API
This is probably due to you running a script, which causes the base color routines to think that they are not being run interactively and this shut off the color. I tried to fix this a while back, but the codepath taken when running a script is a little convoluted. On Aug 22, 2014 6:33 PM, Spencer Russell spencer.f.russ...@gmail.com wrote: I just filed an issue [1] that currently the print_with_color() doesn't seem to use the correct colors on my system. While checking it out, I was thinking that print_with_color might fit in nicely with Color.jl. We could pull print_with_color out of Base and add all the color terminal codes to Color.jl, which could implement print(msg::String, color::ColorValue) and related functions. This feels more in line with Julian style of leveraging the type system to replace functions with underscores. [1]: https://github.com/JuliaLang/julia/issues/8096
