[julia-users] Re: Manual deallocation
In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
[julia-users] Default Methods for abstract objects - overriding getfield
Suppose I have an abstract class. I want to provide a kind of default
property of all sub-types of it.
I believe I can do this using a function, for example
abstract Machine
function data(m::Machine)
Int[]
end
type BigMachine : Machine
data::Vector{Integer}
end
function data(m::BigMachine) m.data end
type SmallMachine : Machine
end
data(BigMachine([1,2,3]))
data(SmallMachine())
This provides the intended default behavour. My belief is that this will
be more space efficient since I am not storing with each machine an empty
data vector.
Is there a penalty in lookup?
It's also not ideal, I did it his way because I cannot overload the
getfield function. The problems here are now that 1. I can't use the dot
notation to access a field 2. For each non-default machine type I have to
define a new accessor function. I suppose I could use a type-union, but it
seems unnecessarily complicated
Is there an idiomatic solution to this?
[julia-users] Re: Default Methods for abstract objects - overriding getfield
Hi Zenna,
there has been a long discussion in
https://github.com/JuliaLang/julia/issues/4935 on the subject. Please have
a look at te whole thread as the thinking has changed quite a bit.
In short you can also just assume that an asbtract type has certain field.
Then you would only have to write the data function one time.
If you are looking for overloading getfield, here is the relevant issue:
https://github.com/JuliaLang/julia/issues/1974
Cheers,
Tobias
Am Dienstag, 8. Juli 2014 09:39:08 UTC+2 schrieb Zenna Tavares:
Suppose I have an abstract class. I want to provide a kind of default
property of all sub-types of it.
I believe I can do this using a function, for example
abstract Machine
function data(m::Machine)
Int[]
end
type BigMachine : Machine
data::Vector{Integer}
end
function data(m::BigMachine) m.data end
type SmallMachine : Machine
end
data(BigMachine([1,2,3]))
data(SmallMachine())
This provides the intended default behavour. My belief is that this will
be more space efficient since I am not storing with each machine an empty
data vector.
Is there a penalty in lookup?
It's also not ideal, I did it his way because I cannot overload the
getfield function. The problems here are now that 1. I can't use the dot
notation to access a field 2. For each non-default machine type I have to
define a new accessor function. I suppose I could use a type-union, but it
seems unnecessarily complicated
Is there an idiomatic solution to this?
[julia-users] julia 0.3 prerelease, Windows, problems to start?
d:\addprogram\Julia 0.3.0-prereleasejulia.lnk ERROR: stat: connection reset by peer (ECONNRESET) d:\addprogram\Julia 0.3.0-prereleasejulia.lnk _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type help() to list help topics | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.3.0-prerelease+3121 (2014-05-19 22:16 UTC) _/ |\__'_|_|_|\__'_| | Commit e1468d5* (49 days old master) |__/ | i686-w64-mingw32 julia I'm trying to run julia on my work laptop (so behind a few company firewalls). In the above copy you see two different situations: 1) i'm connected directly to the Internet. julia doesn't start with an error message. 2) i'm connected via the company net. julia starts. Which is nice, but after that i cannot use Pkg.update etc. (see https://groups.google.com/forum/?fromgroups=#!topic/julia-users/zjTrgs8M3x4). Any hint? Wishing a happy day, Andreas
[julia-users] Re: Question about 'dot' notation (and max/maximum)
What has happened, two months later, to the (in)famous 'dot' notation in
Julia?
There was such a convincing discussion that 5 + x shall not be correct, so
I got used to it. When I now by chance try
julia x = [0.1, 0.2, 0.3];
julia 5 + x
3-element Array{Float64,1}:
5.1
5.2
5.3
julia versioninfo()
Julia Version 0.3.0-prerelease+3921
Commit 0b46af5* (2014-06-28 02:01 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: Intel(R) Core(TM) i3-3217U CPU @ 1.80GHz
WORD_SIZE: 64
BLAS: libblas.so.3
LAPACK: liblapack.so.3
LIBM: libopenlibm
[ PS: This was the last PPA update I could get on Ubuntu. ]
On Friday, May 2, 2014 3:55:20 PM UTC+2, Hans W Borchers wrote:
I have to admit that I am quite unhappy with some of the changed features
in
Julia version 0.3.0, especially the 'dot' notation. Here are some examples.
Let x be a vector defined as x = [0.1, 0.2, 0.3] . Then typing
julia 5 + x
WARNING: x::Number + A::Array is deprecated, use x .+ A instead.
Re: [julia-users] Re: Question about 'dot' notation (and max/maximum)
Le mardi 08 juillet 2014 à 01:51 -0700, Hans W Borchers a écrit :
What has happened, two months later, to the (in)famous 'dot' notation
in Julia?
There was such a convincing discussion that 5 + x shall not be
correct, so
I got used to it. When I now by chance try
julia x = [0.1, 0.2, 0.3];
julia 5 + x
3-element Array{Float64,1}:
5.1
5.2
5.3
julia versioninfo()
Julia Version 0.3.0-prerelease+3921
Commit 0b46af5* (2014-06-28 02:01 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: Intel(R) Core(TM) i3-3217U CPU @ 1.80GHz
WORD_SIZE: 64
BLAS: libblas.so.3
LAPACK: liblapack.so.3
LIBM: libopenlibm
[ PS: This was the last PPA update I could get on Ubuntu. ]
See https://github.com/JuliaLang/julia/pull/7226
Regards
[julia-users] Re: Using Sparse Matrix Algorithms: Spatial Econometrics
Hi Don, I think the reason they're not sparse is that in the line A = (In - rho*W); the matrix In is not sparse: if you replace the line In = eye(n); by In = speye(n); the result should then be sparse. However at the moment there doesn't seem to be a sparse det method (I just filed issue #7543 https://github.com/JuliaLang/julia/issues/7543): you can get around this by using log(det(lufact(A))) instead of log(det(A)). You might also want to have a quick look at the performance tips http://docs.julialang.org/en/latest/manual/performance-tips/, in particular: * You should use randn() instead of randn(1): randn() samples a single Float64 scalar, randn(1) samples a vector of length 1. On my machine (using 0.3 pre-release) your code doesn't work, as this promotes some vectors to matrices, which doesn't work with dot. You also then won't need to call variables like rho_proposed[1]. * Try wrapping everything in a function: this makes it easier for the JIT compiler to work its magic, as well as making it easier to use profiling tools. * If you're using dense matrices, you can use logdet(A) instead of log(det(A)): this can avoid some underflow and overflow problems (unfortunately this doesn't work for sparse lu factorizations yet, see the issue mentioned above). * Purely from a style point of view, there's no need to keep functions in separate files, or end lines with semi-colons. Simon On Tuesday, 8 July 2014 01:48:56 UTC+1, Donald Lacombe wrote: Dear Julia Users, I am currently developing/converting several Bayesian spatial econometric models from MATLAB into Julia. I have successfully coded the spatial autoregressive model (SAR) with diffuse priors in Julia but have a question regarding the use of sparse matrix algorithms. The models use a spatial weight matrix which is usually sparse in nature and this matrix appears in many of the expressions, especially in the random walk Metropolis algorithm used to obtain the marginal distribution for the spatial autocorrelation parameter rho. Here is a code snippet and the complete code is attached: # Draw for rho A = (In - rho*W); denominator = log(det(A))-.5*sigma^-2*dot(A*y-x*beta,A*y-x*beta); accept = 0; rho_proposed = rho + zta*randn(1); while accept == 0 if ((rho_proposed[1] -1) (rho_proposed[1] 1)); accept = 1; else rho_proposed = rho + zta*randn(1); end end B = (In - rho_proposed[1]*W); numerator = log(det(B))-.5*sigma^-2*dot(B*y-x*beta,B*y-x*beta); u = rand(1); if ((numerator[1] - denominator[1]) exp(1)) pp = 1; else ratio = exp(numerator[1] - denominator[1]); pp = min(1,ratio); end if (u[1] pp[1]) rho = rho_proposed[1]; acc = acc + 1; end ar = acc/gibbs; if ar 0.4 zta = zta/1.1; end if ar 0.6 zta = zta*1.1; end A = (In - rho*W); Basically, I'm wondering if there is any way to make the A and B matrices sparse and possibly make it run faster, especially in the log(det(A)) terms. Currently, 110 draws (with 10 burn) takes approximately 8 seconds on my 64 bit, core i7 laptop. The computational speed decreases with the sample size n because the weight matrices are treated as full. Any help would be greatly appreciated and if anyone is interested in running the code, the Distributions and Distance packages must be included and initiated first. Regards, Don
Re: [julia-users] Re: Using Sparse Matrix Algorithms: Spatial Econometrics
I think I-ρW would be even better in this case. I is a generic identity matrix. A and B are probably positive definite. If so, I think logdet(cholfact(A)) would be the best solution. 2014-07-08 11:06 GMT+02:00 Simon Byrne simonby...@gmail.com: Hi Don, I think the reason they're not sparse is that in the line A = (In - rho*W); the matrix In is not sparse: if you replace the line In = eye(n); by In = speye(n); the result should then be sparse. However at the moment there doesn't seem to be a sparse det method (I just filed issue #7543 https://github.com/JuliaLang/julia/issues/7543): you can get around this by using log(det(lufact(A))) instead of log(det(A)). You might also want to have a quick look at the performance tips http://docs.julialang.org/en/latest/manual/performance-tips/, in particular: * You should use randn() instead of randn(1): randn() samples a single Float64 scalar, randn(1) samples a vector of length 1. On my machine (using 0.3 pre-release) your code doesn't work, as this promotes some vectors to matrices, which doesn't work with dot. You also then won't need to call variables like rho_proposed[1]. * Try wrapping everything in a function: this makes it easier for the JIT compiler to work its magic, as well as making it easier to use profiling tools. * If you're using dense matrices, you can use logdet(A) instead of log(det(A)): this can avoid some underflow and overflow problems (unfortunately this doesn't work for sparse lu factorizations yet, see the issue mentioned above). * Purely from a style point of view, there's no need to keep functions in separate files, or end lines with semi-colons. Simon On Tuesday, 8 July 2014 01:48:56 UTC+1, Donald Lacombe wrote: Dear Julia Users, I am currently developing/converting several Bayesian spatial econometric models from MATLAB into Julia. I have successfully coded the spatial autoregressive model (SAR) with diffuse priors in Julia but have a question regarding the use of sparse matrix algorithms. The models use a spatial weight matrix which is usually sparse in nature and this matrix appears in many of the expressions, especially in the random walk Metropolis algorithm used to obtain the marginal distribution for the spatial autocorrelation parameter rho. Here is a code snippet and the complete code is attached: # Draw for rho A = (In - rho*W); denominator = log(det(A))-.5*sigma^-2*dot(A*y-x*beta,A*y-x*beta); accept = 0; rho_proposed = rho + zta*randn(1); while accept == 0 if ((rho_proposed[1] -1) (rho_proposed[1] 1)); accept = 1; else rho_proposed = rho + zta*randn(1); end end B = (In - rho_proposed[1]*W); numerator = log(det(B))-.5*sigma^-2*dot(B*y-x*beta,B*y-x*beta); u = rand(1); if ((numerator[1] - denominator[1]) exp(1)) pp = 1; else ratio = exp(numerator[1] - denominator[1]); pp = min(1,ratio); end if (u[1] pp[1]) rho = rho_proposed[1]; acc = acc + 1; end ar = acc/gibbs; if ar 0.4 zta = zta/1.1; end if ar 0.6 zta = zta*1.1; end A = (In - rho*W); Basically, I'm wondering if there is any way to make the A and B matrices sparse and possibly make it run faster, especially in the log(det(A)) terms. Currently, 110 draws (with 10 burn) takes approximately 8 seconds on my 64 bit, core i7 laptop. The computational speed decreases with the sample size n because the weight matrices are treated as full. Any help would be greatly appreciated and if anyone is interested in running the code, the Distributions and Distance packages must be included and initiated first. Regards, Don -- Med venlig hilsen Andreas Noack Jensen
[julia-users] Re: Question about 'dot' notation (and max/maximum)
Has this been announced somewhere? (Frankly,I'm not reading julia-dev on a regular basis.) I see that the manual does not reflects this change. See section Vectorized Operators and Functions: Some operators without dots operate elementwise anyway when one argument is a scalar. These operators are *, /, \, and the bitwise operators. though .+, .- are not in the list of binary arithmetic operators anymore (and do work properly). And actually, 1 / [1.0, 2.0] does now operate elementwise, but gives a deprecated warning once. Will it be deprecated, or stay alive as the manual claims. Dear core developers, don't listen to Julia newbies (like me) too closely -- stick to what you think is reasonable. On Tuesday, July 8, 2014 10:51:30 AM UTC+2, Hans W Borchers wrote: What has happened, two months later, to the (in)famous 'dot' notation in Julia? There was such a convincing discussion that 5 + x shall not be correct, so I got used to it. When I now by chance try ... On Friday, May 2, 2014 3:55:20 PM UTC+2, Hans W Borchers wrote: I have to admit that I am quite unhappy with some of the changed features in Julia version 0.3.0, especially the 'dot' notation. Here are some examples. Let x be a vector defined as x = [0.1, 0.2, 0.3] . Then typing julia 5 + x WARNING: x::Number + A::Array is deprecated, use x .+ A instead.
[julia-users] Re: Question about 'dot' notation (and max/maximum)
Just commenting on the lack of anouncement. As all this happend on a developement branch of Julia which is always a little in flux these things should not really be anounced. What counts is how the NEWS file looks in the end. Am Dienstag, 8. Juli 2014 11:56:12 UTC+2 schrieb Hans W Borchers: Has this been announced somewhere? (Frankly,I'm not reading julia-dev on a regular basis.) I see that the manual does not reflects this change. See section Vectorized Operators and Functions: Some operators without dots operate elementwise anyway when one argument is a scalar. These operators are *, /, \, and the bitwise operators. though .+, .- are not in the list of binary arithmetic operators anymore (and do work properly). And actually, 1 / [1.0, 2.0] does now operate elementwise, but gives a deprecated warning once. Will it be deprecated, or stay alive as the manual claims. Dear core developers, don't listen to Julia newbies (like me) too closely -- stick to what you think is reasonable. On Tuesday, July 8, 2014 10:51:30 AM UTC+2, Hans W Borchers wrote: What has happened, two months later, to the (in)famous 'dot' notation in Julia? There was such a convincing discussion that 5 + x shall not be correct, so I got used to it. When I now by chance try ... On Friday, May 2, 2014 3:55:20 PM UTC+2, Hans W Borchers wrote: I have to admit that I am quite unhappy with some of the changed features in Julia version 0.3.0, especially the 'dot' notation. Here are some examples. Let x be a vector defined as x = [0.1, 0.2, 0.3] . Then typing julia 5 + x WARNING: x::Number + A::Array is deprecated, use x .+ A instead.
[julia-users] Re: julia 0.3 prerelease, Windows, problems to start?
Try one of the newer builds (last week or less). This issue was addressed in https://github.com/JuliaLang/julia/issues/5574 On Tuesday, July 8, 2014 4:47:50 AM UTC-4, Andreas Lobinger wrote: d:\addprogram\Julia 0.3.0-prereleasejulia.lnk ERROR: stat: connection reset by peer (ECONNRESET) d:\addprogram\Julia 0.3.0-prereleasejulia.lnk _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type help() to list help topics | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.3.0-prerelease+3121 (2014-05-19 22:16 UTC) _/ |\__'_|_|_|\__'_| | Commit e1468d5* (49 days old master) |__/ | i686-w64-mingw32 julia I'm trying to run julia on my work laptop (so behind a few company firewalls). In the above copy you see two different situations: 1) i'm connected directly to the Internet. julia doesn't start with an error message. 2) i'm connected via the company net. julia starts. Which is nice, but after that i cannot use Pkg.update etc. (see https://groups.google.com/forum/?fromgroups=#!topic/julia-users/zjTrgs8M3x4 ). Any hint? Wishing a happy day, Andreas
[julia-users] Re: julia 0.3 prerelease, Windows, problems to start?
Little bit better, but still not there... On Tuesday, July 8, 2014 11:26:40 AM UTC+2, Willy Heineman wrote: Try one of the newer builds (last week or less). This issue was addressed in https://github.com/JuliaLang/julia/issues/5574 D:\addprogram\Julia-0.3.0-prereleasejulia.lnk _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type help() to list help topics | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.3.0-prerelease+4081 (2014-07-08 00:42 UTC) _/ |\__'_|_|_|\__'_| | Commit 8856c8a (0 days old master) |__/ | i686-w64-mingw32 ERROR: stat: connection reset by peer (ECONNRESET) D:\addprogram\Julia-0.3.0-prerelease
Re: [julia-users] Re: Question about 'dot' notation (and max/maximum)
Fully realising that this discussion has been settled and the convention is here to stay, I nevertheless feel obsessed to make the remark that there would have been more elegant solutions. Other languages have been able to come up with acceptable operators for a binary 'min' or 'max': https://gcc.gnu.org/onlinedocs/gcc-2.95.2/gcc_5.html#SEC107 (less elegant alternative since the difference between min and max is confusing: http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/viewer.htm#langref_sect10.htm ) The linguistic difference between max and maximum is very subtle, maybe because I am not a native English speaker. From the mathematical point of view, I think of a maximum (but preferably abbreviated as max) always in the context of a function or a set, never in the context of the largest out of two numbers (not really a reference or definition, but http://en.wikipedia.org/wiki/Maxima_and_minima only/directly discusses functions and sets). ? compares to max as + compares to sum would have been so much more convincing :-). That's all, enough bikeshedding for now, back to work. Op maandag 5 mei 2014 17:46:05 UTC+2 schreef Stefan Karpinski: I think at this point, while the maximum name may not be everyone's favorite choice, that bikeshed has sailed. Unless we're going to solve this problem differently, it's going to stay the way it is: max is to maximum as + is to sum. On Mon, May 5, 2014 at 11:39 AM, Ethan Anderes ethana...@gmail.com javascript: wrote: +1 for max() and pmax(). It seems the easiest to write and to interpret.
Re: [julia-users] Question about 'dot' notation (and max/maximum)
Since I just read these operators were later removed from gcc again, it must not all have been perfect either :D. On 08 Jul 2014, at 16:04, Jutho juthohaege...@gmail.com wrote: Fully realising that this discussion has been settled and the convention is here to stay, I nevertheless feel obsessed to make the remark that there would have been more elegant solutions. Other languages have been able to come up with acceptable operators for a binary 'min' or 'max': https://gcc.gnu.org/onlinedocs/gcc-2.95.2/gcc_5.html#SEC107 (less elegant alternative since the difference between min and max is confusing: http://support.sas.com/documentation/cdl/en/imlug/59656/HTML/default/viewer.htm#langref_sect10.htm ) The linguistic difference between max and maximum is very subtle, maybe because I am not a native English speaker. From the mathematical point of view, I think of a maximum (but preferably abbreviated as max) always in the context of a function or a set, never in the context of the largest out of two numbers (not really a reference or definition, but http://en.wikipedia.org/wiki/Maxima_and_minima only/directly discusses functions and sets). ? compares to max as + compares to sum would have been so much more convincing :-). That's all, enough bikeshedding for now, back to work. Op maandag 5 mei 2014 17:46:05 UTC+2 schreef Stefan Karpinski: I think at this point, while the maximum name may not be everyone's favorite choice, that bikeshed has sailed. Unless we're going to solve this problem differently, it's going to stay the way it is: max is to maximum as + is to sum. On Mon, May 5, 2014 at 11:39 AM, Ethan Anderes ethana...@gmail.com wrote: +1 for max() and pmax(). It seems the easiest to write and to interpret.
[julia-users] Contributing an external package
Hi everybody, I wrote a vanilla feedforward neural network package a bit ago and was hoping work to polish it up (with the long term goal of getting it registered with the Julia distribution). The package repo can be found here: https://github.com/EricChiang/ANN.jl Can anyone provide general tips (or links) regarding standards for package development, particularly for ML models? This includes testing, academic citations, general API construction, etc. Thanks! Eric
Re: [julia-users] Re: julia 0.3 prerelease, Windows, problems to start?
Please open a github issue. What is your %HOMEDRIVE%%HOMEPATH%? On Tue, Jul 8, 2014 at 9:47 AM, Andreas Lobinger lobing...@gmail.com wrote: Little bit better, but still not there... On Tuesday, July 8, 2014 11:26:40 AM UTC+2, Willy Heineman wrote: Try one of the newer builds (last week or less). This issue was addressed in https://github.com/JuliaLang/julia/issues/5574 D:\addprogram\Julia-0.3.0-prereleasejulia.lnk _ _ _ _(_)_ | A fresh approach to technical computing (_) | (_) (_)| Documentation: http://docs.julialang.org _ _ _| |_ __ _ | Type help() to list help topics | | | | | | |/ _` | | | | |_| | | | (_| | | Version 0.3.0-prerelease+4081 (2014-07-08 00:42 UTC) _/ |\__'_|_|_|\__'_| | Commit 8856c8a (0 days old master) |__/ | i686-w64-mingw32 ERROR: stat: connection reset by peer (ECONNRESET) D:\addprogram\Julia-0.3.0-prerelease
Re: [julia-users] Re: Manual deallocation
Are there any plans to document the Julia GC (i.e. your comment implies that it's a stop-the-world GC) and/or add performance tuning knobs? On Tue, Jul 8, 2014 at 12:17 PM, Stefan Karpinski ste...@karpinski.org wrote: Writing `A = nothing` in Julia will not cause the memory used by A to be freed immediately. That happens in reference counted systems, which many dynamic languages traditionally have been, but which Julia is not. Instead, the memory for A will be freed the next time a garbage collection occurs. This consists of the language runtime stopping everything it's doing, tracing through the graph of all objects in memory, marking the ones it can still reach, and freeing all the rest. So if doing `A = nothing` causes there to be no more reachable references to the object that A used to point at, then that object will be freed when the next garbage collection occurs. Normally, garbage collection occurs automatically when the system tries to allocate something and doesn't have enough memory to do so: it runs the garbage collector and then tries again. You can, however, call gc() to force garbage collection to occur now. This is generally not necessary or recommended. On Mon, Jul 7, 2014 at 11:04 PM, Ivar Nesje iva...@gmail.com wrote: In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
[julia-users] Gadfly and Jewel: label colors
In terms of rich output in the recent release I really just threw together the most basic thing that worked. For now that means there're going to be some issues like this, but the good news is I'll be adding lots of frills and polish for the next release. Ideally I do want to support both light and dark themes as seamlessly as possible, so long as it doesn't turn it to be to tricky.
Re: [julia-users] Re: Manual deallocation
It is currently, stop-the-world, although that could change. What kind of documentation would you be interested in? There's really nothing interesting about Julia's GC at this point – it's completely vanilla stop-the-world, mark-and-sweep. On Tue, Jul 8, 2014 at 9:28 AM, Abraham Egnor abe.eg...@gmail.com wrote: Are there any plans to document the Julia GC (i.e. your comment implies that it's a stop-the-world GC) and/or add performance tuning knobs? On Tue, Jul 8, 2014 at 12:17 PM, Stefan Karpinski ste...@karpinski.org wrote: Writing `A = nothing` in Julia will not cause the memory used by A to be freed immediately. That happens in reference counted systems, which many dynamic languages traditionally have been, but which Julia is not. Instead, the memory for A will be freed the next time a garbage collection occurs. This consists of the language runtime stopping everything it's doing, tracing through the graph of all objects in memory, marking the ones it can still reach, and freeing all the rest. So if doing `A = nothing` causes there to be no more reachable references to the object that A used to point at, then that object will be freed when the next garbage collection occurs. Normally, garbage collection occurs automatically when the system tries to allocate something and doesn't have enough memory to do so: it runs the garbage collector and then tries again. You can, however, call gc() to force garbage collection to occur now. This is generally not necessary or recommended. On Mon, Jul 7, 2014 at 11:04 PM, Ivar Nesje iva...@gmail.com wrote: In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
[julia-users] Re: Understanding how to display an Image and change pixels in real-time (fast)
Ok , I know what I am going to do :).
So I have split my projet into 4 parts...
*Part 1:*
I need to find a way to store pixels in a 3D Euclidean space. (X,Y,Z) with
X,Y and Z *INTEGER*
I can do this with a 3xXxYxZ Array{Uint8,4}.
But I will need to know if a (X,Y,Z) position is empty or not (fast) and
Ultimatly I woud like to have only the list of the pixels that exist really
( if the (X, Y, Z) position is empty I don't need to store this information
into my data)
*Part 2:*I need a fonction who will take two arguments and give me a
picture.
Like this:
#pos is the position (x,y,z) of the camera and vec is the direction (x,y,z)
looking by the camera
function camera_view(pos,vec)
#explore the data from *part 1* and return a 3x600x800
Array{Uint8,3} (for a 800x600 resolution screen)end
*Part 3:*
For this part what i need is to create a loop that display and refresh on
the screen the returned picture of the function from
*part 2*Something like this:
img = imread(mypicture.bmp)
display(img, pixelspacing = [1,1])
when (camera is moving)
img.data = camera_view(pos,vec)
display(img, pixelspacing = [1,1])
end
*Part 4:*Use the keybord and the mouse to trigger the loop from *part 3*
and use the right (pos and vec) in the camera_view(pos,vec) function
*-The end--*I
create this topic to answer the part3
so lets forget about all the other part.
My question transform to:
*HOW CAN I DISPLAY x in a windows or using all the screenlike
this:LOOP x = **rand(Uint8,3,800,600)*
*Display (x)*
*END*
and how can I refresh my windows/screen with a new x the fastest way
* (the point here is that I want to make a movie with random bitmaps)*idealy
without using any Package/library (writting all the code lines from
nothing)
I am doing this to LEARN I know there are clever programmers outhere who
made perfect tools for 3D.
I will post my work when I will get closer to the end haha.
Thanks again
[julia-users] Re: ANN: Processing.jl
Animations are there; check out the demo. I've decided to go for simplicity and let the user wrap animations in loops, which should call the animate() function to update the screen at the end of every frame, rather than building a draw() function that automatically and perpetually loops. This actually allows for more a bit more flexibility over the animation than the typical Processing approach allows. I've also abandoned the setup() function, which isn't necessary here. Since this is being built on Tk (an eventual option to request Gtk will be added, once functionality stabilises), this also means that we do have interactivity. I'll be making that easier for users today, wrapping everything in Processing friendly functions. Basic support for images, text, and spatial transformations will also be updated later today. After that, I plan to make animations a bit faster, by supporting OS specific drawing surfaces, rather than the general Cairo surface. Once the 2D support is stabilised through Cairo + Tk/Gtk, an OpenGL based backend for 2D drawing will be added. Best, Rob On Monday, July 7, 2014 3:53:39 PM UTC+2, Job van der Zwan wrote: Cool! As someone who uses Processing to prototypes pretty much everything, I'll have a look. One important feature is that it makes super easy to set up an interactive loop - your description makes it sound like it doesn't do that (yet). Of course, the biggest draw to Processing is that it has third party libraries for just about anything you could possibly http://processing.org/reference/libraries/ want a library for as an artist. It will be a while before Julia is at that point ;) On Sunday, 6 July 2014 18:09:02 UTC+2, Robert Ennis wrote: Hey everyone, The first basic, but usable, version of Processing.jl is up on METADATA.jl as a non-tagged package. To try it out, do a Pkg.clone(Processing). So far, you can replicate some of the basic 2D drawing functionality of the original Processing environment and script some basic animations by playing with colours. You will find a basic example in the test directory of the package. 3D support is on hold until the 2D support is finalised. There is nothing special about this package. It's just a small, convenience wrapper around the amazing work from the people who have put together the Tk.jl, Cairo.jl (w/ Pango), and Color.jl packages. Thanks for an awesome community and awesome work to build on! :) Best, Rob
[julia-users] Re: ANN: Processing.jl
And, yes, Julia packages in the art department are still few and young, but we're getting there. ;) Best, Rob On Tuesday, July 8, 2014 6:48:25 PM UTC+2, Robert Ennis wrote: Animations are there; check out the demo. I've decided to go for simplicity and let the user wrap animations in loops, which should call the animate() function to update the screen at the end of every frame, rather than building a draw() function that automatically and perpetually loops. This actually allows for more a bit more flexibility over the animation than the typical Processing approach allows. I've also abandoned the setup() function, which isn't necessary here. Since this is being built on Tk (an eventual option to request Gtk will be added, once functionality stabilises), this also means that we do have interactivity. I'll be making that easier for users today, wrapping everything in Processing friendly functions. Basic support for images, text, and spatial transformations will also be updated later today. After that, I plan to make animations a bit faster, by supporting OS specific drawing surfaces, rather than the general Cairo surface. Once the 2D support is stabilised through Cairo + Tk/Gtk, an OpenGL based backend for 2D drawing will be added. Best, Rob On Monday, July 7, 2014 3:53:39 PM UTC+2, Job van der Zwan wrote: Cool! As someone who uses Processing to prototypes pretty much everything, I'll have a look. One important feature is that it makes super easy to set up an interactive loop - your description makes it sound like it doesn't do that (yet). Of course, the biggest draw to Processing is that it has third party libraries for just about anything you could possibly http://processing.org/reference/libraries/ want a library for as an artist. It will be a while before Julia is at that point ;) On Sunday, 6 July 2014 18:09:02 UTC+2, Robert Ennis wrote: Hey everyone, The first basic, but usable, version of Processing.jl is up on METADATA.jl as a non-tagged package. To try it out, do a Pkg.clone(Processing). So far, you can replicate some of the basic 2D drawing functionality of the original Processing environment and script some basic animations by playing with colours. You will find a basic example in the test directory of the package. 3D support is on hold until the 2D support is finalised. There is nothing special about this package. It's just a small, convenience wrapper around the amazing work from the people who have put together the Tk.jl, Cairo.jl (w/ Pango), and Color.jl packages. Thanks for an awesome community and awesome work to build on! :) Best, Rob
[julia-users] julia in emacs path
Hi all, I'm trying to get julia working via ESS in Aquamacs 3.0a. I managed to get it running on one computer but can't seem to get it on another one with very similar setups. The first thing I did was add this to my .emacs (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) I also tried (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) Also, in the instructions here https://github.com/emacs-ess/ESS/wiki/Julia it says to also have (load path/to/ESS/git/lisp/ess-site) or (require 'ess-site) in .emacs. But both of these return the following error with the newer versions of Aquamacs/Emacs. I assume this is because ESS is now pre-packaged with emacs distributions but am not sure: File error: Cannot open load file, no such file or directory, ess-site ESS is working with R still on this computer, so it does seem to be loaded. With either the julia or julia-basic version above, M-x julia doesn't work (emacs says [No match]). I then tried to start julia from a shell started in the terminal. It didn't work because julia didn't seem to be in my path. So I added julia to my PATH in .bash_profile and can stat it now by typing 'julia' in shell now from the terminal. But if I start shell in Emacs, it still doesn't work. I figured out that this was because shell in emacs doesn't run .bash_profile automatically. So I added (setenv PATH (shell-command-to-string source ~/.bash_profile; echo -n $PATH)) to my .emacs. But then I get this error: bash-3.2$ julia ERROR: could not start process `tput setaf 0`: no such file or directory ( ENOENT) in test_success at process.jl:460 in success at process.jl:468 in _start at client.jl:417 Process shell exited abnormally with code 1 I'm not very experienced with .emacs or .bash_profiles or PATH variables. So any help would be much appreciated! Thanks, Steve
Re: [julia-users] Re: Manual deallocation
You may also find this pull request interesting: https://github.com/JuliaLang/julia/pull/5227 On Tuesday, July 8, 2014 11:31:12 AM UTC-5, Abraham Egnor wrote: Are there any plans to document the Julia GC (i.e. your comment implies that it's a stop-the-world GC) and/or add performance tuning knobs? On Tue, Jul 8, 2014 at 12:17 PM, Stefan Karpinski ste...@karpinski.org wrote: Writing `A = nothing` in Julia will not cause the memory used by A to be freed immediately. That happens in reference counted systems, which many dynamic languages traditionally have been, but which Julia is not. Instead, the memory for A will be freed the next time a garbage collection occurs. This consists of the language runtime stopping everything it's doing, tracing through the graph of all objects in memory, marking the ones it can still reach, and freeing all the rest. So if doing `A = nothing` causes there to be no more reachable references to the object that A used to point at, then that object will be freed when the next garbage collection occurs. Normally, garbage collection occurs automatically when the system tries to allocate something and doesn't have enough memory to do so: it runs the garbage collector and then tries again. You can, however, call gc() to force garbage collection to occur now. This is generally not necessary or recommended. On Mon, Jul 7, 2014 at 11:04 PM, Ivar Nesje iva...@gmail.com wrote: In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
Re: [julia-users] Re: Understanding how to display an Image and change pixels in real-time (fast)
For 3D, Simon is the go-to person.
But for just displaying an image (which was your question at the bottom),
ImageView will do exactly what you want. See the README, focusing especially
on the programmatic usage section. If you're storing the array color-first,
you'll need to wrap it in an Image. See the Images package.
--Tim
On Tuesday, July 08, 2014 09:45:04 AM De Prins Maxime wrote:
Ok , I know what I am going to do :).
So I have split my projet into 4 parts...
*Part 1:*
I need to find a way to store pixels in a 3D Euclidean space. (X,Y,Z) with
X,Y and Z *INTEGER*
I can do this with a 3xXxYxZ Array{Uint8,4}.
But I will need to know if a (X,Y,Z) position is empty or not (fast) and
Ultimatly I woud like to have only the list of the pixels that exist really
( if the (X, Y, Z) position is empty I don't need to store this information
into my data)
*Part 2:*I need a fonction who will take two arguments and give me a
picture.
Like this:
#pos is the position (x,y,z) of the camera and vec is the direction (x,y,z)
looking by the camera
function camera_view(pos,vec)
#explore the data from *part 1* and return a 3x600x800
Array{Uint8,3} (for a 800x600 resolution screen)end
*Part 3:*
For this part what i need is to create a loop that display and refresh on
the screen the returned picture of the function from
*part 2*Something like this:
img = imread(mypicture.bmp)
display(img, pixelspacing = [1,1])
when (camera is moving)
img.data = camera_view(pos,vec)
display(img, pixelspacing = [1,1])
end
*Part 4:*Use the keybord and the mouse to trigger the loop from *part 3*
and use the right (pos and vec) in the camera_view(pos,vec) function
*-The end--*I
create this topic to answer the part3
so lets forget about all the other part.
My question transform to:
*HOW CAN I DISPLAY x in a windows or using all the screenlike
this:LOOP x = **rand(Uint8,3,800,600)*
*Display (x)*
*END*
and how can I refresh my windows/screen with a new x the fastest way
* (the point here is that I want to make a movie with random bitmaps)*idealy
without using any Package/library (writting all the code lines from
nothing)
I am doing this to LEARN I know there are clever programmers outhere who
made perfect tools for 3D.
I will post my work when I will get closer to the end haha.
Thanks again
Re: [julia-users] Re: julia 0.3 prerelease, Windows, problems to start?
Thanks for the question. On Tuesday, July 8, 2014 4:38:15 PM UTC+2, Jameson wrote: Please open a github issue. What is your %HOMEDRIVE%%HOMEPATH%? afaics (i'm not really a windows commandline user, i use cygwin for that) HOMEPATH=\ HOMEDRIVE=H: and as you can assume, H: is only available if i'm connected to the company network. Afaics the only julia specific in H: is the .julia_history while a previous installation .julia/v0.3 package dir resides below Users/ApplicationData/. How can i choose the directories at installation time (to make sure H: isn't used)?
[julia-users] Re: parallel_until?
Could you file an issue about the segfault with the code to reproduce it? Definitely not useful if it segfaults. -viral On Monday, July 7, 2014 12:45:38 AM UTC-7, Robert Feldt wrote: BTW, can anyone explain why it segfaults if one continues increasing the number of procs available: feldt:~/tmp$ julia03 -p 32 test_pfind_colwise.jl 0.05507445327586207 feldt:~/tmp$ julia03 -p 64 test_pfind_colwise.jl 0.0358273456 /Users/feldt/feldt/bin/julia03: line 2: 68828 Segmentation fault: 11 /Applications/Julia-0.3.0-prerelease-3e6a6c7bd8.app/Contents/Resources/julia/bin/julia $1 $2 $3 $4 $5 $6 $7 $8 $9 Still a speedup though but not useable if it segfaults... ;) Cheers, /Robert Den måndagen den 7:e juli 2014 kl. 09:40:40 UTC+2 skrev Robert Feldt: If someone else needs something like this I ended up copying and modifying pmap to suit my needs. I'm sure it can be simplified and shortened but it works. Example code below and I get healthy speedups all the way up to 20 procs: feldt:~/tmp$ julia03 test_pfind_colwise.jl 0.5952118622068966 feldt:~/tmp$ julia03 -p 2 test_pfind_colwise.jl 0.3214775730689655 feldt:~/tmp$ julia03 -p 4 test_pfind_colwise.jl 0.1769931201724138 feldt:~/tmp$ julia03 -p 8 test_pfind_colwise.jl 0.10350024568965517 feldt:~/tmp$ julia03 -p 16 test_pfind_colwise.jl 0.0708839620001 feldt:~/tmp$ julia03 -p 32 test_pfind_colwise.jl 0.05338699896551725 Julia sure is great! :) Cheers, Robert N = 100 Rows = 3 MaxValue = 10 @everywhere function fake_slow_condition(x) sleep(0.01) sum(x) == 30 ? x : false # Return x iff it fulfills the poperty end # Similar to Base.findfirst(predicate, A) but process the array A columnwise in parallel # and opt out of further processing if found. Returns both the index at which found # as well as the result returned. @everywhere function pfind_colwise(predicate, A) np = nprocs() # determine the number of processes available n = size(A, 2) i = 1 results = Any[false for i in 1:n] found = false wasfound(idx) = found = true isfound() = found # function to produce the next work item from the queue. # in this case it's just an index to a column. nextidx() = (idx=i; i+=1; idx) @sync begin for p=1:np if p != myid() || np == 1 @async begin while true idx = nextidx() if idx n break end results[idx] = remotecall_fetch(p, predicate, A[:,idx]) if isfound() break # Someone else already found it end if results[idx] != false wasfound(idx) break # We found it end end end end end end fidx = findfirst((r)-(r != false), results) (fidx, (fidx != 0 ? results[fidx] : nothing)) end Reps = 30 times = zeros(Reps) for rep in 1:Reps array = ifloor(MaxValue * rand(Rows, N)) # We will be looking for the first column that sums to 30. Make sure it exists. suma = sum(array, 1) i = findfirst(suma, 30) if i == 0 # Put it somewhere around the middle i = ifloor(rand(0.45N:0.55*N)) array[:,i] = MaxValue * ones(Int64, Rows, 1) end tic() idx, res = pfind_colwise(fake_slow_condition, array) times[rep] = toq() end println(mean(times[2:Reps]))
[julia-users] Re: Contributing an external package
Have you seen this section of the manual: http://docs.julialang.org/en/latest/manual/packages/ ? -viral On Tuesday, July 8, 2014 7:29:03 AM UTC-7, Eric Chiang wrote: Hi everybody, I wrote a vanilla feedforward neural network package a bit ago and was hoping work to polish it up (with the long term goal of getting it registered with the Julia distribution). The package repo can be found here: https://github.com/EricChiang/ANN.jl Can anyone provide general tips (or links) regarding standards for package development, particularly for ML models? This includes testing, academic citations, general API construction, etc. Thanks! Eric
[julia-users] @profile and memory allocation
I believe I saw discussion of finding areas of memory allocation in the output of Profile.view() or the flame plot from ProfileView.view(). However, like so many other things, I fail to remember the details. The particular case I am interested in is described in an IJulia notebook called Bootstrap.ipynb in the repository github.com/dmbates/JuliaWorkshop. I have eliminated almost all of the memory allocation in the inner loop of a simulation but there is some vestigial allocation going on that I can't track down. The notebook can be viewed at http://nbviewer.ipython.org/github/dmbates/JuliaWorkshop/blob/master/Bootstrap.ipynb
Re: [julia-users] @profile and memory allocation
You can find the places where garbage is collected using ProfileView; bars in red correspond to lines that trigger gc. For this to work properly, your C backtraces have to resolve the function name correctly. That worked for me on Kubuntu 12.04, but after upgrading to Kubuntu 14.04 now I get garbage back from C backtraces. (Go figure.) In that case, I recommend saying ProfileView.view(C=true) and then at least you can guess that it's a gc event from the tall stack of bars (the garbage-collector is heavily recursive). Of course, garbage collection does not always happen at every line that triggers allocation. You may also be interested in https://github.com/JuliaLang/julia/pull/7464 which more directly detects allocation. It should work on your machine as long as you don't make install julia in a separate directory from where you cloned it (that proves to be the source of the Travis failure). --Tim On Tuesday, July 08, 2014 11:22:46 AM Douglas Bates wrote: I believe I saw discussion of finding areas of memory allocation in the output of Profile.view() or the flame plot from ProfileView.view(). However, like so many other things, I fail to remember the details. The particular case I am interested in is described in an IJulia notebook called Bootstrap.ipynb in the repository github.com/dmbates/JuliaWorkshop. I have eliminated almost all of the memory allocation in the inner loop of a simulation but there is some vestigial allocation going on that I can't track down. The notebook can be viewed at http://nbviewer.ipython.org/github/dmbates/JuliaWorkshop/blob/master/Bootstr ap.ipynb
[julia-users] Re: Understanding how to display an Image and change pixels in real-time (fast)
Thanks tim, I made a wonderfull movie. but you need to have a 800*600 image.bmp to work properly using Images using ImageView img = imread(image.bmp) c = canvasgrid(1,1) ops = [:pixelspacing = [1,1]] display(c[1,1], img; ops...) for i = 1:200 img.data = rand(Uint8,3,800,600) display(c[1,1], img; ops...) end This is enought to start thinking about part1 and part2. But i don't understand what happening in my computer :( How to do this without using any package (What code lines of the images and imageview package I use) , i will dig inside. Max
Re: [julia-users] Passing parameters to 'optimize'
This should work, except that you're not providing any initial values for x. Is your problem one-dimensional or multi-dimensional? -- John On Jul 8, 2014, at 11:44 AM, Mathieu Taschereau-Dumouchel mat...@gmail.com wrote: I am trying to optimize a simple function that takes a parameter as an argument. Something like: a=2 f(x)=-(x-a)^2 optimize(f) Is there any way to do this? I have played quite a bit with optimize without success. Thanks a lot Mathieu
[julia-users] Passing parameters to 'optimize'
I am trying to optimize a simple function that takes a parameter as an argument. Something like: a=2 f(x)=-(x-a)^2 optimize(f) Is there any way to do this? I have played quite a bit with optimize without success. Thanks a lot Mathieu
Re: [julia-users] julia in emacs path
Thanks for that. I installed the package but it's still not functioning. Interestingly, I have noticed that I neither start R nor Julia from an Emacs shell. And again, although M-x R does work M-x julia does not. Other ideas? On Tuesday, July 8, 2014 12:29:31 PM UTC-5, Ista Zahn wrote: Hi Steve, I'm guess that at least part of the problem is that OSX doesn't set environment variables nicely, making it hard for emacs to find things. Try installing https://github.com/purcell/exec-path-from-shell in emacs, restart, and see if that does the trick. Best, Ista On Tue, Jul 8, 2014 at 1:00 PM, Steve Bellan steve@gmail.com javascript: wrote: Hi all, I'm trying to get julia working via ESS in Aquamacs 3.0a. I managed to get it running on one computer but can't seem to get it on another one with very similar setups. The first thing I did was add this to my .emacs (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) I also tried (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) Also, in the instructions here it says to also have (load path/to/ESS/git/lisp/ess-site) or (require 'ess-site) in .emacs. But both of these return the following error with the newer versions of Aquamacs/Emacs. I assume this is because ESS is now pre-packaged with emacs distributions but am not sure: File error: Cannot open load file, no such file or directory, ess-site ESS is working with R still on this computer, so it does seem to be loaded. With either the julia or julia-basic version above, M-x julia doesn't work (emacs says [No match]). I then tried to start julia from a shell started in the terminal. It didn't work because julia didn't seem to be in my path. So I added julia to my PATH in .bash_profile and can stat it now by typing 'julia' in shell now from the terminal. But if I start shell in Emacs, it still doesn't work. I figured out that this was because shell in emacs doesn't run .bash_profile automatically. So I added (setenv PATH (shell-command-to-string source ~/.bash_profile; echo -n $PATH)) to my .emacs. But then I get this error: bash-3.2$ julia ERROR: could not start process `tput setaf 0`: no such file or directory (ENOENT) in test_success at process.jl:460 in success at process.jl:468 in _start at client.jl:417 Process shell exited abnormally with code 1 I'm not very experienced with .emacs or .bash_profiles or PATH variables. So any help would be much appreciated! Thanks, Steve
Re: [julia-users] Passing parameters to 'optimize'
Thanks for the quick reply! My problem is one-dimensional. Here is the code: using Optim a=2 f(x)=-(x-a)^2 optimize(f,3) I get: ERROR: no method optimize(Function, Int64) I am running Version 0.3.0-prerelease+3884 (2014-06-25 10:41 UTC) and have updated all packages. Thanks for your help Mathieu On Tuesday, July 8, 2014 2:48:24 PM UTC-4, John Myles White wrote: This should work, except that you're not providing any initial values for x. Is your problem one-dimensional or multi-dimensional? -- John On Jul 8, 2014, at 11:44 AM, Mathieu Taschereau-Dumouchel mat...@gmail.com javascript: wrote: I am trying to optimize a simple function that takes a parameter as an argument. Something like: a=2 f(x)=-(x-a)^2 optimize(f) Is there any way to do this? I have played quite a bit with optimize without success. Thanks a lot Mathieu
Re: [julia-users] julia in emacs path
scratch that, R does start up in the emacs shell. And I fixed my path variable so I can start julia up in the emacs shell too (had a syntax error in my .bash_profile). What I still cannot get is M-x julia to work on emacs start-up. However, M-x julia does work once I've run M-x R. It seems like M-x R is loading ESS somehow? However, once I get julia open with M-x julia, Julia is basically frozen. If I C-g, then it unfreezes but seems to be a read-only buffer with no working julia implementation.. On Tuesday, July 8, 2014 1:55:02 PM UTC-5, Steve Bellan wrote: Thanks for that. I installed the package but it's still not functioning. Interestingly, I have noticed that I neither start R nor Julia from an Emacs shell. And again, although M-x R does work M-x julia does not. Other ideas? On Tuesday, July 8, 2014 12:29:31 PM UTC-5, Ista Zahn wrote: Hi Steve, I'm guess that at least part of the problem is that OSX doesn't set environment variables nicely, making it hard for emacs to find things. Try installing https://github.com/purcell/exec-path-from-shell in emacs, restart, and see if that does the trick. Best, Ista On Tue, Jul 8, 2014 at 1:00 PM, Steve Bellan steve@gmail.com wrote: Hi all, I'm trying to get julia working via ESS in Aquamacs 3.0a. I managed to get it running on one computer but can't seem to get it on another one with very similar setups. The first thing I did was add this to my .emacs (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) I also tried (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) Also, in the instructions here it says to also have (load path/to/ESS/git/lisp/ess-site) or (require 'ess-site) in .emacs. But both of these return the following error with the newer versions of Aquamacs/Emacs. I assume this is because ESS is now pre-packaged with emacs distributions but am not sure: File error: Cannot open load file, no such file or directory, ess-site ESS is working with R still on this computer, so it does seem to be loaded. With either the julia or julia-basic version above, M-x julia doesn't work (emacs says [No match]). I then tried to start julia from a shell started in the terminal. It didn't work because julia didn't seem to be in my path. So I added julia to my PATH in .bash_profile and can stat it now by typing 'julia' in shell now from the terminal. But if I start shell in Emacs, it still doesn't work. I figured out that this was because shell in emacs doesn't run .bash_profile automatically. So I added (setenv PATH (shell-command-to-string source ~/.bash_profile; echo -n $PATH)) to my .emacs. But then I get this error: bash-3.2$ julia ERROR: could not start process `tput setaf 0`: no such file or directory (ENOENT) in test_success at process.jl:460 in success at process.jl:468 in _start at client.jl:417 Process shell exited abnormally with code 1 I'm not very experienced with .emacs or .bash_profiles or PATH variables. So any help would be much appreciated! Thanks, Steve
[julia-users] Need help on Sublime-IJulia install on Windows, Julia 0.3.0-prerelease
I run Julia 0.3.0-prerelease on Windows 7 64. I am trying to install Sublime-IJulia that I would like to evaluate, looks like exactly what I am looking for as an IDE. I went through all the install steps https://github.com/quinnj/Sublime-IJulia. I keep on getting the dreaded ***Kernel Died***. I suspect same issue as github #53 https://github.com/quinnj/Sublime-IJulia/issues/53 I commented on as well. Does anyone have an idea on how to move forward? For instance, how could I find out the root cause or try a more recent Julia build on windows? Thanks.
Re: [julia-users] julia in emacs path
OK, got it working after more playing. I had to remove the path from my .emacs file get it write in the .bash_profile. It seemed that the path was set more than once in emacs bash and this was causing the crash and ess not recognizing it. On Tuesday, July 8, 2014 2:06:35 PM UTC-5, Steve Bellan wrote: scratch that, R does start up in the emacs shell. And I fixed my path variable so I can start julia up in the emacs shell too (had a syntax error in my .bash_profile). What I still cannot get is M-x julia to work on emacs start-up. However, M-x julia does work once I've run M-x R. It seems like M-x R is loading ESS somehow? However, once I get julia open with M-x julia, Julia is basically frozen. If I C-g, then it unfreezes but seems to be a read-only buffer with no working julia implementation.. On Tuesday, July 8, 2014 1:55:02 PM UTC-5, Steve Bellan wrote: Thanks for that. I installed the package but it's still not functioning. Interestingly, I have noticed that I neither start R nor Julia from an Emacs shell. And again, although M-x R does work M-x julia does not. Other ideas? On Tuesday, July 8, 2014 12:29:31 PM UTC-5, Ista Zahn wrote: Hi Steve, I'm guess that at least part of the problem is that OSX doesn't set environment variables nicely, making it hard for emacs to find things. Try installing https://github.com/purcell/exec-path-from-shell in emacs, restart, and see if that does the trick. Best, Ista On Tue, Jul 8, 2014 at 1:00 PM, Steve Bellan steve@gmail.com wrote: Hi all, I'm trying to get julia working via ESS in Aquamacs 3.0a. I managed to get it running on one computer but can't seem to get it on another one with very similar setups. The first thing I did was add this to my .emacs (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) I also tried (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) Also, in the instructions here it says to also have (load path/to/ESS/git/lisp/ess-site) or (require 'ess-site) in .emacs. But both of these return the following error with the newer versions of Aquamacs/Emacs. I assume this is because ESS is now pre-packaged with emacs distributions but am not sure: File error: Cannot open load file, no such file or directory, ess-site ESS is working with R still on this computer, so it does seem to be loaded. With either the julia or julia-basic version above, M-x julia doesn't work (emacs says [No match]). I then tried to start julia from a shell started in the terminal. It didn't work because julia didn't seem to be in my path. So I added julia to my PATH in .bash_profile and can stat it now by typing 'julia' in shell now from the terminal. But if I start shell in Emacs, it still doesn't work. I figured out that this was because shell in emacs doesn't run .bash_profile automatically. So I added (setenv PATH (shell-command-to-string source ~/.bash_profile; echo -n $PATH)) to my .emacs. But then I get this error: bash-3.2$ julia ERROR: could not start process `tput setaf 0`: no such file or directory (ENOENT) in test_success at process.jl:460 in success at process.jl:468 in _start at client.jl:417 Process shell exited abnormally with code 1 I'm not very experienced with .emacs or .bash_profiles or PATH variables. So any help would be much appreciated! Thanks, Steve
Re: [julia-users] Re: Manual deallocation
Mostly I was looking for documentation on performance characteristics, although I'd settle for anything at all :) The Julia home page and manual have zero mention of memory management that I could find, except for a FAQ entry about unbinding large values in the REPL. I inferred GC but the runtime could also have been using refcounting, region inference, or something more exotic. On Tue, Jul 8, 2014 at 1:26 PM, Patrick O'Leary patrick.ole...@gmail.com wrote: You may also find this pull request interesting: https://github.com/JuliaLang/julia/pull/5227 On Tuesday, July 8, 2014 11:31:12 AM UTC-5, Abraham Egnor wrote: Are there any plans to document the Julia GC (i.e. your comment implies that it's a stop-the-world GC) and/or add performance tuning knobs? On Tue, Jul 8, 2014 at 12:17 PM, Stefan Karpinski ste...@karpinski.org wrote: Writing `A = nothing` in Julia will not cause the memory used by A to be freed immediately. That happens in reference counted systems, which many dynamic languages traditionally have been, but which Julia is not. Instead, the memory for A will be freed the next time a garbage collection occurs. This consists of the language runtime stopping everything it's doing, tracing through the graph of all objects in memory, marking the ones it can still reach, and freeing all the rest. So if doing `A = nothing` causes there to be no more reachable references to the object that A used to point at, then that object will be freed when the next garbage collection occurs. Normally, garbage collection occurs automatically when the system tries to allocate something and doesn't have enough memory to do so: it runs the garbage collector and then tries again. You can, however, call gc() to force garbage collection to occur now. This is generally not necessary or recommended. On Mon, Jul 7, 2014 at 11:04 PM, Ivar Nesje iva...@gmail.com wrote: In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
Re: [julia-users] julia in emacs path
Hi Steve, Glad you got it working. I just tried it out with aquamacs (can't stand it myself, but to each his own...) and it worked for me, provided that I set (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) rather than (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) Best, Ista On Tue, Jul 8, 2014 at 3:25 PM, Steve Bellan steve.bel...@gmail.com wrote: OK, got it working after more playing. I had to remove the path from my .emacs file get it write in the .bash_profile. It seemed that the path was set more than once in emacs bash and this was causing the crash and ess not recognizing it. On Tuesday, July 8, 2014 2:06:35 PM UTC-5, Steve Bellan wrote: scratch that, R does start up in the emacs shell. And I fixed my path variable so I can start julia up in the emacs shell too (had a syntax error in my .bash_profile). What I still cannot get is M-x julia to work on emacs start-up. However, M-x julia does work once I've run M-x R. It seems like M-x R is loading ESS somehow? However, once I get julia open with M-x julia, Julia is basically frozen. If I C-g, then it unfreezes but seems to be a read-only buffer with no working julia implementation.. On Tuesday, July 8, 2014 1:55:02 PM UTC-5, Steve Bellan wrote: Thanks for that. I installed the package but it's still not functioning. Interestingly, I have noticed that I neither start R nor Julia from an Emacs shell. And again, although M-x R does work M-x julia does not. Other ideas? On Tuesday, July 8, 2014 12:29:31 PM UTC-5, Ista Zahn wrote: Hi Steve, I'm guess that at least part of the problem is that OSX doesn't set environment variables nicely, making it hard for emacs to find things. Try installing https://github.com/purcell/exec-path-from-shell in emacs, restart, and see if that does the trick. Best, Ista On Tue, Jul 8, 2014 at 1:00 PM, Steve Bellan steve@gmail.com wrote: Hi all, I'm trying to get julia working via ESS in Aquamacs 3.0a. I managed to get it running on one computer but can't seem to get it on another one with very similar setups. The first thing I did was add this to my .emacs (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia) I also tried (setq inferior-julia-program-name /Applications/Julia-0.2.1.app/Contents/Resources/julia/bin/julia-basic) Also, in the instructions here it says to also have (load path/to/ESS/git/lisp/ess-site) or (require 'ess-site) in .emacs. But both of these return the following error with the newer versions of Aquamacs/Emacs. I assume this is because ESS is now pre-packaged with emacs distributions but am not sure: File error: Cannot open load file, no such file or directory, ess-site ESS is working with R still on this computer, so it does seem to be loaded. With either the julia or julia-basic version above, M-x julia doesn't work (emacs says [No match]). I then tried to start julia from a shell started in the terminal. It didn't work because julia didn't seem to be in my path. So I added julia to my PATH in .bash_profile and can stat it now by typing 'julia' in shell now from the terminal. But if I start shell in Emacs, it still doesn't work. I figured out that this was because shell in emacs doesn't run .bash_profile automatically. So I added (setenv PATH (shell-command-to-string source ~/.bash_profile; echo -n $PATH)) to my .emacs. But then I get this error: bash-3.2$ julia ERROR: could not start process `tput setaf 0`: no such file or directory (ENOENT) in test_success at process.jl:460 in success at process.jl:468 in _start at client.jl:417 Process shell exited abnormally with code 1 I'm not very experienced with .emacs or .bash_profiles or PATH variables. So any help would be much appreciated! Thanks, Steve
[julia-users] Re: Using Sparse Matrix Algorithms: Spatial Econometrics
Dear Simon, Thank you for the quick and informative response! I have a few questions/observations: 1) Thank you for the information regarding the randn() function. I found a workaround in the original code but now it looks much better and acts as expected. 2) When I use A = sparse(In - rho*W) I get the following warning from Julia and a break: julia sar_gibbs LoadError(C:/Users/Don/Desktop/Julia Materials/SAR Gibbs/sar_gibbs.jl,13,ErrorException(The inverse of a sparse matrix can often be dense and can cause the computer to run out of memory. If you are sure you have enough memory, please convert your matrix to a dense matrix.)) Apparently, it does not like it when I invert the A matrix. 3) Can I wrap the code with a blank function call like so: function sar_gibbs()? 4) Are you indicating that I can include the knn_weights_matrix code in the same file? Once again, thank you for the response. My goal is to produce a suite of Bayesian spatial econometric models for public consumption and you help is greatly appreciated. Regards, Don On Tuesday, July 8, 2014 5:06:14 AM UTC-4, Simon Byrne wrote: Hi Don, I think the reason they're not sparse is that in the line A = (In - rho*W); the matrix In is not sparse: if you replace the line In = eye(n); by In = speye(n); the result should then be sparse. However at the moment there doesn't seem to be a sparse det method (I just filed issue #7543 https://github.com/JuliaLang/julia/issues/7543): you can get around this by using log(det(lufact(A))) instead of log(det(A)). You might also want to have a quick look at the performance tips http://docs.julialang.org/en/latest/manual/performance-tips/, in particular: * You should use randn() instead of randn(1): randn() samples a single Float64 scalar, randn(1) samples a vector of length 1. On my machine (using 0.3 pre-release) your code doesn't work, as this promotes some vectors to matrices, which doesn't work with dot. You also then won't need to call variables like rho_proposed[1]. * Try wrapping everything in a function: this makes it easier for the JIT compiler to work its magic, as well as making it easier to use profiling tools. * If you're using dense matrices, you can use logdet(A) instead of log(det(A)): this can avoid some underflow and overflow problems (unfortunately this doesn't work for sparse lu factorizations yet, see the issue mentioned above). * Purely from a style point of view, there's no need to keep functions in separate files, or end lines with semi-colons. Simon On Tuesday, 8 July 2014 01:48:56 UTC+1, Donald Lacombe wrote: Dear Julia Users, I am currently developing/converting several Bayesian spatial econometric models from MATLAB into Julia. I have successfully coded the spatial autoregressive model (SAR) with diffuse priors in Julia but have a question regarding the use of sparse matrix algorithms. The models use a spatial weight matrix which is usually sparse in nature and this matrix appears in many of the expressions, especially in the random walk Metropolis algorithm used to obtain the marginal distribution for the spatial autocorrelation parameter rho. Here is a code snippet and the complete code is attached: # Draw for rho A = (In - rho*W); denominator = log(det(A))-.5*sigma^-2*dot(A*y-x*beta,A*y-x*beta); accept = 0; rho_proposed = rho + zta*randn(1); while accept == 0 if ((rho_proposed[1] -1) (rho_proposed[1] 1)); accept = 1; else rho_proposed = rho + zta*randn(1); end end B = (In - rho_proposed[1]*W); numerator = log(det(B))-.5*sigma^-2*dot(B*y-x*beta,B*y-x*beta); u = rand(1); if ((numerator[1] - denominator[1]) exp(1)) pp = 1; else ratio = exp(numerator[1] - denominator[1]); pp = min(1,ratio); end if (u[1] pp[1]) rho = rho_proposed[1]; acc = acc + 1; end ar = acc/gibbs; if ar 0.4 zta = zta/1.1; end if ar 0.6 zta = zta*1.1; end A = (In - rho*W); Basically, I'm wondering if there is any way to make the A and B matrices sparse and possibly make it run faster, especially in the log(det(A)) terms. Currently, 110 draws (with 10 burn) takes approximately 8 seconds on my 64 bit, core i7 laptop. The computational speed decreases with the sample size n because the weight matrices are treated as full. Any help would be greatly appreciated and if anyone is interested in running the code, the Distributions and Distance packages must be included and initiated first. Regards, Don
Re: [julia-users] Re: Using Sparse Matrix Algorithms: Spatial Econometrics
Andreas, Yes, you are correct and your solution works just fine. Thank you for taking the time to answer my question. I've said it before but the kindness and generosity of this community is remarkable! Thanks, Don On Tuesday, July 8, 2014 5:34:45 AM UTC-4, Andreas Noack wrote: I think I-ρW would be even better in this case. I is a generic identity matrix. A and B are probably positive definite. If so, I think logdet(cholfact(A)) would be the best solution. 2014-07-08 11:06 GMT+02:00 Simon Byrne simon...@gmail.com javascript:: Hi Don, I think the reason they're not sparse is that in the line A = (In - rho*W); the matrix In is not sparse: if you replace the line In = eye(n); by In = speye(n); the result should then be sparse. However at the moment there doesn't seem to be a sparse det method (I just filed issue #7543 https://github.com/JuliaLang/julia/issues/7543): you can get around this by using log(det(lufact(A))) instead of log(det(A)). You might also want to have a quick look at the performance tips http://docs.julialang.org/en/latest/manual/performance-tips/, in particular: * You should use randn() instead of randn(1): randn() samples a single Float64 scalar, randn(1) samples a vector of length 1. On my machine (using 0.3 pre-release) your code doesn't work, as this promotes some vectors to matrices, which doesn't work with dot. You also then won't need to call variables like rho_proposed[1]. * Try wrapping everything in a function: this makes it easier for the JIT compiler to work its magic, as well as making it easier to use profiling tools. * If you're using dense matrices, you can use logdet(A) instead of log(det(A)): this can avoid some underflow and overflow problems (unfortunately this doesn't work for sparse lu factorizations yet, see the issue mentioned above). * Purely from a style point of view, there's no need to keep functions in separate files, or end lines with semi-colons. Simon On Tuesday, 8 July 2014 01:48:56 UTC+1, Donald Lacombe wrote: Dear Julia Users, I am currently developing/converting several Bayesian spatial econometric models from MATLAB into Julia. I have successfully coded the spatial autoregressive model (SAR) with diffuse priors in Julia but have a question regarding the use of sparse matrix algorithms. The models use a spatial weight matrix which is usually sparse in nature and this matrix appears in many of the expressions, especially in the random walk Metropolis algorithm used to obtain the marginal distribution for the spatial autocorrelation parameter rho. Here is a code snippet and the complete code is attached: # Draw for rho A = (In - rho*W); denominator = log(det(A))-.5*sigma^-2*dot(A*y-x*beta,A*y-x*beta); accept = 0; rho_proposed = rho + zta*randn(1); while accept == 0 if ((rho_proposed[1] -1) (rho_proposed[1] 1)); accept = 1; else rho_proposed = rho + zta*randn(1); end end B = (In - rho_proposed[1]*W); numerator = log(det(B))-.5*sigma^-2*dot(B*y-x*beta,B*y-x*beta); u = rand(1); if ((numerator[1] - denominator[1]) exp(1)) pp = 1; else ratio = exp(numerator[1] - denominator[1]); pp = min(1,ratio); end if (u[1] pp[1]) rho = rho_proposed[1]; acc = acc + 1; end ar = acc/gibbs; if ar 0.4 zta = zta/1.1; end if ar 0.6 zta = zta*1.1; end A = (In - rho*W); Basically, I'm wondering if there is any way to make the A and B matrices sparse and possibly make it run faster, especially in the log(det(A)) terms. Currently, 110 draws (with 10 burn) takes approximately 8 seconds on my 64 bit, core i7 laptop. The computational speed decreases with the sample size n because the weight matrices are treated as full. Any help would be greatly appreciated and if anyone is interested in running the code, the Distributions and Distance packages must be included and initiated first. Regards, Don -- Med venlig hilsen Andreas Noack Jensen
Re: [julia-users] Re: Manual deallocation
Yes, we should probably have a manual chapter on memory management. On Tue, Jul 8, 2014 at 12:46 PM, Abraham Egnor abe.eg...@gmail.com wrote: Mostly I was looking for documentation on performance characteristics, although I'd settle for anything at all :) The Julia home page and manual have zero mention of memory management that I could find, except for a FAQ entry about unbinding large values in the REPL. I inferred GC but the runtime could also have been using refcounting, region inference, or something more exotic. On Tue, Jul 8, 2014 at 1:26 PM, Patrick O'Leary patrick.ole...@gmail.com wrote: You may also find this pull request interesting: https://github.com/JuliaLang/julia/pull/5227 On Tuesday, July 8, 2014 11:31:12 AM UTC-5, Abraham Egnor wrote: Are there any plans to document the Julia GC (i.e. your comment implies that it's a stop-the-world GC) and/or add performance tuning knobs? On Tue, Jul 8, 2014 at 12:17 PM, Stefan Karpinski ste...@karpinski.org wrote: Writing `A = nothing` in Julia will not cause the memory used by A to be freed immediately. That happens in reference counted systems, which many dynamic languages traditionally have been, but which Julia is not. Instead, the memory for A will be freed the next time a garbage collection occurs. This consists of the language runtime stopping everything it's doing, tracing through the graph of all objects in memory, marking the ones it can still reach, and freeing all the rest. So if doing `A = nothing` causes there to be no more reachable references to the object that A used to point at, then that object will be freed when the next garbage collection occurs. Normally, garbage collection occurs automatically when the system tries to allocate something and doesn't have enough memory to do so: it runs the garbage collector and then tries again. You can, however, call gc() to force garbage collection to occur now. This is generally not necessary or recommended. On Mon, Jul 7, 2014 at 11:04 PM, Ivar Nesje iva...@gmail.com wrote: In julia we don't say you shouldn't do something that could give better performance (if you really want it). The thing is that Julia uses automatic garbage collection because it is a pain to do manually, and then you have to live with the semantics of a garbage collector. If your program is not really constrained by memory in the second part, I would guess that it is unlikely that it would matter to your program when the arrays are released. Freeing memory in julia (and other GC based languages), is about ensuring that no references remains to the allocated object. If it is a global variable, you can assign `nothing`, and if it is a global constant, you can change the type, so you must reassign it to a smaller array with the same dimensionality and type and ensure that you don't have local variables that references the same array. If it is a local variable, I'm not sure there is other options than to arrange the function boundaries, so that the large array goes out of scope when it is not needed any more. kl. 22:56:35 UTC+2 mandag 7. juli 2014 skrev Pablo Zubieta følgende: Let's say I have some large matrices I need to do some calculations and I use them in order to get some results that I will use in a second part of a computation where I not longer need the initial matrices. Suppose also that I preallocate those matrices. Would it be ok to bind the names of those matrices to nothing (or something similar) from the moment I won't be using them anymore, or should I leave the deallocation work to the GC?
Re: [julia-users] Passing parameters to 'optimize'
Ok. Then there are two problems witht the inputs you're using: (1) One-dimensional optimize doesn't take a single starting point, it takes a bracket in which you think the minimum occurs. (2) You need to use floating point inputs. Things will work if you do optimize(f, -10.0, 10.0) or something similar. -- John On Jul 8, 2014, at 12:03 PM, Mathieu Taschereau-Dumouchel mat...@gmail.com wrote: Thanks for the quick reply! My problem is one-dimensional. Here is the code: using Optim a=2 f(x)=-(x-a)^2 optimize(f,3) I get: ERROR: no method optimize(Function, Int64) I am running Version 0.3.0-prerelease+3884 (2014-06-25 10:41 UTC) and have updated all packages. Thanks for your help Mathieu On Tuesday, July 8, 2014 2:48:24 PM UTC-4, John Myles White wrote: This should work, except that you're not providing any initial values for x. Is your problem one-dimensional or multi-dimensional? -- John On Jul 8, 2014, at 11:44 AM, Mathieu Taschereau-Dumouchel mat...@gmail.com wrote: I am trying to optimize a simple function that takes a parameter as an argument. Something like: a=2 f(x)=-(x-a)^2 optimize(f) Is there any way to do this? I have played quite a bit with optimize without success. Thanks a lot Mathieu
Re: [julia-users] Types and interfaces
Definitely a missing piece. Here's the relevant issue: https://github.com/JuliaLang/julia/issues/6975. It's largely a question of design and implementation. On Tue, Jul 8, 2014 at 12:56 PM, Abraham Egnor abe.eg...@gmail.com wrote: I'm very new to Julia, so my apologies for the bits I inevitably get wrong. As far as I can tell, Julia has no notion of interface, i.e. if you declare your data type a subtype of Foo, you also must/should implement these functions on the type. This seems like a pretty significant lack to me - it turns the (otherwise quite lovely) type system into essentially tagged duck typing. There are a few packages that implement their own version of interfaces (and it speaks well of Julia that this is possible!), but they do so with very different semantics and don't expose any sort of general interface-construction machinery. Are there any proposals for an interface framework as part of the standard library?
Re: [julia-users] Passing parameters to 'optimize'
Ahh I see. It works now. Thanks a lot! Mathieu On Tuesday, July 8, 2014 4:00:44 PM UTC-4, John Myles White wrote: Ok. Then there are two problems witht the inputs you're using: (1) One-dimensional optimize doesn't take a single starting point, it takes a bracket in which you think the minimum occurs. (2) You need to use floating point inputs. Things will work if you do optimize(f, -10.0, 10.0) or something similar. -- John On Jul 8, 2014, at 12:03 PM, Mathieu Taschereau-Dumouchel mat...@gmail.com javascript: wrote: Thanks for the quick reply! My problem is one-dimensional. Here is the code: using Optim a=2 f(x)=-(x-a)^2 optimize(f,3) I get: ERROR: no method optimize(Function, Int64) I am running Version 0.3.0-prerelease+3884 (2014-06-25 10:41 UTC) and have updated all packages. Thanks for your help Mathieu On Tuesday, July 8, 2014 2:48:24 PM UTC-4, John Myles White wrote: This should work, except that you're not providing any initial values for x. Is your problem one-dimensional or multi-dimensional? -- John On Jul 8, 2014, at 11:44 AM, Mathieu Taschereau-Dumouchel mat...@gmail.com wrote: I am trying to optimize a simple function that takes a parameter as an argument. Something like: a=2 f(x)=-(x-a)^2 optimize(f) Is there any way to do this? I have played quite a bit with optimize without success. Thanks a lot Mathieu
Re: [julia-users] How do you do interactive development in Julia?
Hi Rob, Thanks for your answer. I use very similar approach with Emacs and ESS mode[1]: I've bound command ess-eval-line-and-step to C-d (CONTROL / COMMAND + D) to be able to send single line and ess-load-file to C-c C-c (double CONTROL / COMMAND + C) to reload the entire file. I don't like idea of reloading REPL, though. Not because it takes 5 commands (I can actually automate most of these keystrokes), but because this way you lose your current state. This may be really painful, especially when your computations took half an hour or so (which is often the case in image processing, recommendation engines, etc.) I think, though, that your ideas combined with clear() function ([2]) from Tim's comment should work pretty well. [1]: https://github.com/emacs-ess/ESS/wiki/Julia [2]: https://github.com/JuliaLang/julia/pull/7487 https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fpull%2F7487sa=Dsntz=1usg=AFQjCNEoxeApYEIgOlcR_uWjEDZtOieb8A
[julia-users] efficient iterative updating for dataframe subsets
Hi, I am writing an MCMC likelihood function that comes from a dynamic
model. I have about 50 state variables for several thousand individuals
that each get updated iteratively over time. However, some individuals'
variables get updated at some type steps but not at others (because they
are inactive). I'd like to be able to write something of the sort
k = 50
n = 1000
states = rand(n, k)
typeof!(states, DataFrame)
function dynupdate(states, active)
with(states[active,:], {
x1 = x1 .* 2
x2 = par1 .* x3 .*x 2
x3 = par3 .* (x1 + x3)^2})
return(states)
}
Forgive the sloppy coding, I'm quite new to Julia. I'm not sure the best
way to do this. The other complication is that I want x1,x2,x3 to all
depend on their value at the last time period. If I update x1 first, then
x2 will depend on the updated version of x1 and not the past one. I could
obviously create x1new, x2new, x3new and then update them in states at the
end, but I'm wondering whats the fastest, most elegant way to do this.
Also, I am struggling to understand how to navigate function documentation
in Julia. When I type help(with) it gives me almost no information on that
function. with() is not an easy thing to search for on google either so I
have no idea what with() does. What's the best way to learn more about
functions in julia? In R, I rely on the help() and example() a lot.
Thanks!
Steve
Re: [julia-users] Ideomatic way to write a function that works with a collection of any Number (any subtype of type)
What you're doing isn't a workaround: it's the correct way to do this in the
current version of Julia. There may be shorthand in the future, but this is the
right approach today.
-- John
On Jul 8, 2014, at 2:01 PM, Andrei Zh faithlessfri...@gmail.com wrote:
Here's another question about code style. Let's say I want to write function
inc() that just adds 1 to its (typed) argument. For simple numbers I can
force parameter to be of type Number:
julia function inc(x::Number) x + 1 end
inc (generic function with 1 method)
julia inc(1)
2
julia inc(1.)
2.0
For parametrised collections, however, it doesn't work, since Julia's type
parameters are invariant:
julia function inc(x::Vector{Number}) x + 1 end
inc (generic function with 2 methods)
julia inc([1, 2])
ERROR: no method inc(Array{Int64,1})
julia inc([1., 2.])
ERROR: no method inc(Array{Float64,1})
As a workaround I use parametrized functions, which work just fine:
julia function inc{T : Number}(x::Vector{T}) x + 1 end
inc (generic function with 3 methods)
julia inc([1, 2])
2-element Array{Int64,1}:
2
3
julia inc([1., 2.])
2-element Array{Float64,1}:
2.0
3.0
But since operations on vectors of (any) numbers are so common, I would
expect simpler / shorter way to write them. More generally, I wonder if
there's a better way to write functions with collections parametrized by
abstract classes.
Additional (but closely related) question: is there any run-time overhead for
function arguments with abstract type parameters over several functions with
concrete types? E.g. is writing inc(x::Vector{Int}) and
inc(x::Vector{Float64}) faster than function inc{T :
Number}(x::Vector{T})?
Re: [julia-users] Types and interfaces
Abraham, you might also be interested in the prototype that I have prepared in https://github.com/JuliaLang/julia/pull/7025 While I am also looking forward that anything like this lands in Julia, this is mainly about getting better error messages and code organization. Julia is perfectly usable even with its implicit interfaces that are available now. By the way, in the Base Graphics module there is a mustimplement macro that also can be used to define interfaces. Am Dienstag, 8. Juli 2014 22:02:52 UTC+2 schrieb Stefan Karpinski: Definitely a missing piece. Here's the relevant issue: https://github.com/JuliaLang/julia/issues/6975. It's largely a question of design and implementation. On Tue, Jul 8, 2014 at 12:56 PM, Abraham Egnor abe@gmail.com javascript: wrote: I'm very new to Julia, so my apologies for the bits I inevitably get wrong. As far as I can tell, Julia has no notion of interface, i.e. if you declare your data type a subtype of Foo, you also must/should implement these functions on the type. This seems like a pretty significant lack to me - it turns the (otherwise quite lovely) type system into essentially tagged duck typing. There are a few packages that implement their own version of interfaces (and it speaks well of Julia that this is possible!), but they do so with very different semantics and don't expose any sort of general interface-construction machinery. Are there any proposals for an interface framework as part of the standard library?
Re: [julia-users] @profile and memory allocation
On Tuesday, July 8, 2014 1:41:40 PM UTC-5, Tim Holy wrote: You can find the places where garbage is collected using ProfileView; bars in red correspond to lines that trigger gc. For this to work properly, your C backtraces have to resolve the function name correctly. That worked for me on Kubuntu 12.04, but after upgrading to Kubuntu 14.04 now I get garbage back from C backtraces. (Go figure.) Thanks. I'm not sure that everything is working properly on Ubuntu 14.04/(LLVM 3.5 from svn) for me but the results seem reasonable. The (admittedly small amount of) memory allocation is taking place in randmtzig_randn from librandom.jl. In that case, I recommend saying ProfileView.view(C=true) and then at least you can guess that it's a gc event from the tall stack of bars (the garbage-collector is heavily recursive). I could see that but I didn't remember how to manipulate the graph. The stack was so tall that the lines I was interested in were hard to select with the mouse. Is there a way of zooming in on the window? Of course, garbage collection does not always happen at every line that triggers allocation. You may also be interested in https://github.com/JuliaLang/julia/pull/7464 which more directly detects allocation. It should work on your machine as long as you don't make install julia in a separate directory from where you cloned it (that proves to be the source of the Travis failure). --Tim On Tuesday, July 08, 2014 11:22:46 AM Douglas Bates wrote: I believe I saw discussion of finding areas of memory allocation in the output of Profile.view() or the flame plot from ProfileView.view(). However, like so many other things, I fail to remember the details. The particular case I am interested in is described in an IJulia notebook called Bootstrap.ipynb in the repository github.com/dmbates/JuliaWorkshop. I have eliminated almost all of the memory allocation in the inner loop of a simulation but there is some vestigial allocation going on that I can't track down. The notebook can be viewed at http://nbviewer.ipython.org/github/dmbates/JuliaWorkshop/blob/master/Bootstr ap.ipynb
Re: [julia-users] How do you do interactive development in Julia?
Hi Andrei, I'll have a look at your pointers before further extending the TextMate bundle, particularly [2] looks promising. Thanks, and regards, Rob J. Goedman goed...@icloud.com On Jul 8, 2014, at 2:19 PM, Andrei Zh faithlessfri...@gmail.com wrote: Hi Rob, Thanks for your answer. I use very similar approach with Emacs and ESS mode[1]: I've bound command ess-eval-line-and-step to C-d (CONTROL / COMMAND + D) to be able to send single line and ess-load-file to C-c C-c (double CONTROL / COMMAND + C) to reload the entire file. I don't like idea of reloading REPL, though. Not because it takes 5 commands (I can actually automate most of these keystrokes), but because this way you lose your current state. This may be really painful, especially when your computations took half an hour or so (which is often the case in image processing, recommendation engines, etc.) I think, though, that your ideas combined with clear() function ([2]) from Tim's comment should work pretty well. [1]: https://github.com/emacs-ess/ESS/wiki/Julia [2]: https://github.com/JuliaLang/julia/pull/7487
[julia-users] Re: efficient iterative updating for dataframe subsets
The documentation
here http://julia.readthedocs.org/en/latest/manual/introduction/ works
beautifully, you can search it by keywords, etc. and every function in the
standard library is represented. As for user-defined functions, I'm not
sure what we can do about that except for looking at the code, etc.
Also, I'm not entirely certain what your code is meant to do. It looks
naively like you could work exclusively with arrays, not a DataFrame.
Arrays are much faster and more efficient. It's not *awful* to just say
newStates = zeros(n,k) and then do all your computation using the original
states array, saving the results in newStates. Then say states = newStates
and your ready for the next run.
On Tuesday, July 8, 2014 2:26:16 PM UTC-7, Steve Bellan wrote:
Hi, I am writing an MCMC likelihood function that comes from a dynamic
model. I have about 50 state variables for several thousand individuals
that each get updated iteratively over time. However, some individuals'
variables get updated at some type steps but not at others (because they
are inactive). I'd like to be able to write something of the sort
k = 50
n = 1000
states = rand(n, k)
typeof!(states, DataFrame)
function dynupdate(states, active)
with(states[active,:], {
x1 = x1 .* 2
x2 = par1 .* x3 .*x 2
x3 = par3 .* (x1 + x3)^2})
return(states)
}
Forgive the sloppy coding, I'm quite new to Julia. I'm not sure the best
way to do this. The other complication is that I want x1,x2,x3 to all
depend on their value at the last time period. If I update x1 first, then
x2 will depend on the updated version of x1 and not the past one. I could
obviously create x1new, x2new, x3new and then update them in states at the
end, but I'm wondering whats the fastest, most elegant way to do this.
Also, I am struggling to understand how to navigate function documentation
in Julia. When I type help(with) it gives me almost no information on that
function. with() is not an easy thing to search for on google either so I
have no idea what with() does. What's the best way to learn more about
functions in julia? In R, I rely on the help() and example() a lot.
Thanks!
Steve
Re: [julia-users] @profile and memory allocation
On Tuesday, July 08, 2014 03:24:02 PM Douglas Bates wrote: Is there a way of zooming in on the window? Just click drag. (If you're running from the REPL; doesn't work in IJulia.) Double-click to go back out to the full view. --Tim
[julia-users] Re: Manual deallocation
Thanks for the answers. I agree that a manual chapter would be helpful.
Re: [julia-users] Ideomatic way to write a function that works with a collection of any Number (any subtype of type)
For your additional question: no overhead for the abstract version versus
the two specialized. Don't think of them as types like in C/C++ function
definition, think of them as a filter. Julia will compile a new version for
every type input you put in anyway that is specialized for the type parsed
in.
On Tuesday, July 8, 2014 2:51:41 PM UTC-7, John Myles White wrote:
What you're doing isn't a workaround: it's the correct way to do this in
the current version of Julia. There may be shorthand in the future, but
this is the right approach today.
-- John
On Jul 8, 2014, at 2:01 PM, Andrei Zh faithle...@gmail.com javascript:
wrote:
Here's another question about code style. Let's say I want to write
function inc() that just adds 1 to its (typed) argument. For simple
numbers I can force parameter to be of type Number:
julia function inc(x::Number) x + 1 end
inc (generic function with 1 method)
julia inc(1)
2
julia inc(1.)
2.0
For parametrised collections, however, it doesn't work, since Julia's
type parameters are invariant:
julia function inc(x::Vector{Number}) x + 1 end
inc (generic function with 2 methods)
julia inc([1, 2])
ERROR: no method inc(Array{Int64,1})
julia inc([1., 2.])
ERROR: no method inc(Array{Float64,1})
As a workaround I use parametrized functions, which work just fine:
julia function inc{T : Number}(x::Vector{T}) x + 1 end
inc (generic function with 3 methods)
julia inc([1, 2])
2-element Array{Int64,1}:
2
3
julia inc([1., 2.])
2-element Array{Float64,1}:
2.0
3.0
But since operations on vectors of (any) numbers are so common, I would
expect simpler / shorter way to write them. More generally, I wonder if
there's a better way to write functions with collections parametrized by
abstract classes.
Additional (but closely related) question: is there any run-time
overhead for function arguments with abstract type parameters over several
functions with concrete types? E.g. is writing inc(x::Vector{Int}) and
inc(x::Vector{Float64}) faster than function inc{T :
Number}(x::Vector{T})?
Re: [julia-users] Ideomatic way to write a function that works with a collection of any Number (any subtype of type)
(on a related note, for the same reason type assertions in a method
definition don't actually improve performance)
On Tuesday, July 8, 2014 6:19:51 PM UTC-7, Iain Dunning wrote:
For your additional question: no overhead for the abstract version
versus the two specialized. Don't think of them as types like in C/C++
function definition, think of them as a filter. Julia will compile a new
version for every type input you put in anyway that is specialized for the
type parsed in.
On Tuesday, July 8, 2014 2:51:41 PM UTC-7, John Myles White wrote:
What you're doing isn't a workaround: it's the correct way to do this in
the current version of Julia. There may be shorthand in the future, but
this is the right approach today.
-- John
On Jul 8, 2014, at 2:01 PM, Andrei Zh faithle...@gmail.com wrote:
Here's another question about code style. Let's say I want to write
function inc() that just adds 1 to its (typed) argument. For simple
numbers I can force parameter to be of type Number:
julia function inc(x::Number) x + 1 end
inc (generic function with 1 method)
julia inc(1)
2
julia inc(1.)
2.0
For parametrised collections, however, it doesn't work, since Julia's
type parameters are invariant:
julia function inc(x::Vector{Number}) x + 1 end
inc (generic function with 2 methods)
julia inc([1, 2])
ERROR: no method inc(Array{Int64,1})
julia inc([1., 2.])
ERROR: no method inc(Array{Float64,1})
As a workaround I use parametrized functions, which work just fine:
julia function inc{T : Number}(x::Vector{T}) x + 1 end
inc (generic function with 3 methods)
julia inc([1, 2])
2-element Array{Int64,1}:
2
3
julia inc([1., 2.])
2-element Array{Float64,1}:
2.0
3.0
But since operations on vectors of (any) numbers are so common, I would
expect simpler / shorter way to write them. More generally, I wonder if
there's a better way to write functions with collections parametrized by
abstract classes.
Additional (but closely related) question: is there any run-time
overhead for function arguments with abstract type parameters over several
functions with concrete types? E.g. is writing inc(x::Vector{Int}) and
inc(x::Vector{Float64}) faster than function inc{T :
Number}(x::Vector{T})?
Re: [julia-users] Types and interfaces
As an experiment, I had recently started work on implementing the reverse of this (extracting the implicit interface given a type) in a pull request for astrieanna's TypeCheck.jl package: https://github.com/vtjnash/TypeCheck.jl/commit/b0ebcac4c2a9a3daccd1eeb1b28030b90ae5e2c9 On Tue, Jul 8, 2014 at 5:56 PM, Tobias Knopp tobias.kn...@googlemail.com wrote: Abraham, you might also be interested in the prototype that I have prepared in https://github.com/JuliaLang/julia/pull/7025 While I am also looking forward that anything like this lands in Julia, this is mainly about getting better error messages and code organization. Julia is perfectly usable even with its implicit interfaces that are available now. By the way, in the Base Graphics module there is a mustimplement macro that also can be used to define interfaces. Am Dienstag, 8. Juli 2014 22:02:52 UTC+2 schrieb Stefan Karpinski: Definitely a missing piece. Here's the relevant issue: https://github.com/JuliaLang/julia/issues/6975. It's largely a question of design and implementation. On Tue, Jul 8, 2014 at 12:56 PM, Abraham Egnor abe@gmail.com wrote: I'm very new to Julia, so my apologies for the bits I inevitably get wrong. As far as I can tell, Julia has no notion of interface, i.e. if you declare your data type a subtype of Foo, you also must/should implement these functions on the type. This seems like a pretty significant lack to me - it turns the (otherwise quite lovely) type system into essentially tagged duck typing. There are a few packages that implement their own version of interfaces (and it speaks well of Julia that this is possible!), but they do so with very different semantics and don't expose any sort of general interface-construction machinery. Are there any proposals for an interface framework as part of the standard library?
[julia-users] The Meaning of `const`
The only reference I can find in the manual to the `const` keyword is in
the performance section [1]. Well, that and the `isconst` function[2].
This seems like it would be worth documenting, especially since it behaves
significantly differently than the const that I remember from Java or C++,
which prevented you from changing the value of const variable.
I think I understand now, that `const` in Julia means that the type of the
variable must remain constant. This was very surprising to me.
~~~.jl
julia const x = 10
10
julia x += 1
Warning: redefining constant x
11
julia x
11
julia const arr = Int[1,2,3]
3-element Array{Int64,1}:
1
2
3
julia arr[1] = 13
13
julia arr
3-element Array{Int64,1}:
13
2
3
julia x += 0.5
ERROR: invalid redefinition of constant x
~~~
I'd be happy to add this to the manual, but I'm not sure where it belongs.
Any suggestions?
1:
http://docs.julialang.org/en/latest/manual/performance-tips/?highlight=const#avoid-global-variables
2:
http://docs.julialang.org/en/latest/stdlib/base/?highlight=const#Base.isconst
Re: [julia-users] Types and interfaces
That's a really cool idea. It will be really interesting to see what our implicit interfaces are determined to be. On Tue, Jul 8, 2014 at 7:22 PM, Jameson Nash vtjn...@gmail.com wrote: As an experiment, I had recently started work on implementing the reverse of this (extracting the implicit interface given a type) in a pull request for astrieanna's TypeCheck.jl package: https://github.com/vtjnash/TypeCheck.jl/commit/b0ebcac4c2a9a3daccd1eeb1b28030b90ae5e2c9 On Tue, Jul 8, 2014 at 5:56 PM, Tobias Knopp tobias.kn...@googlemail.com wrote: Abraham, you might also be interested in the prototype that I have prepared in https://github.com/JuliaLang/julia/pull/7025 While I am also looking forward that anything like this lands in Julia, this is mainly about getting better error messages and code organization. Julia is perfectly usable even with its implicit interfaces that are available now. By the way, in the Base Graphics module there is a mustimplement macro that also can be used to define interfaces. Am Dienstag, 8. Juli 2014 22:02:52 UTC+2 schrieb Stefan Karpinski: Definitely a missing piece. Here's the relevant issue: https://github.com/JuliaLang/julia/issues/6975. It's largely a question of design and implementation. On Tue, Jul 8, 2014 at 12:56 PM, Abraham Egnor abe@gmail.com wrote: I'm very new to Julia, so my apologies for the bits I inevitably get wrong. As far as I can tell, Julia has no notion of interface, i.e. if you declare your data type a subtype of Foo, you also must/should implement these functions on the type. This seems like a pretty significant lack to me - it turns the (otherwise quite lovely) type system into essentially tagged duck typing. There are a few packages that implement their own version of interfaces (and it speaks well of Julia that this is possible!), but they do so with very different semantics and don't expose any sort of general interface-construction machinery. Are there any proposals for an interface framework as part of the standard library?
[julia-users] Gaussian process factor analysis
I was wondering if anyone has done any work on Gaussian process factor analysis in julia? I'm interested in the techniques in this paper: 1. Yu, B. M. *et al.* Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. *Journal of Neurophysiology* *102,* 614–635 (2009). There is already a matlab implementation here: http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_Dimensionality_Reduction.html but it would nice to have a pure julia implementatin as well. I'm aware of the discussion here: https://groups.google.com/d/msg/julia-users/aBJIcjwqzCM/vF5-fXPrWhsJ about porting other parts of the same toolbox, so if I'm going to do this myself, I'll work straight from the original paper to avoid licensing issues.
