[julia-users] Re: ClusterManagers timeout

Thanks for the pointer to the offending line Ben! On Wednesday, September 28, 2016 at 4:34:48 PM UTC+1, Ben Arthur wrote: > > you could custom modify your copy of ClusterManagers.jl to wait longer by > changing the hard-coded "60" on this line: > > >

[julia-users] ClusterManagers timeout

Hello If I request some nodes using addprocs_slurm, slurm of course enqueues the request. However, if the request is not fulfilled within a certain time, I get a timeout type message from ClusterManagers. Is there a way of telling ClusterManagers I'm prepared to wait (a long time) for my

[julia-users] Re: Performance tips for network data transfer?

Thanks for the thoughts people, much appreciated, gives me some ideas to work with. I'm going to play around with pure Julia solutions first as my prior experience trying to get MPI.jl running on my cluster in a REPL was painful. This could be the wrong attitude and I may have to change it.

[julia-users] Performance tips for network data transfer?

Dear Julians I'm trying to speed up the network data transfer in an MCMC algorithm. Currently I'm using `remotecall` based functions to do the network communication. Essentially for every node I include I scale up by I incur about 50mb of data transfer per iteration. The topology is that

[julia-users] Error involving LabelNode

I've had a long-running job crash out a couple of times now with an error message I can't fathom: ```ERROR: On worker 2: On worker 2: MethodError: `convert` has no method matching convert(::Type{Type{LabelNode}}, ::Int64) This may have arisen from a call to the constructor Type{LabelNode}(...),

[julia-users] Re: Achitecture for solving largish LASSO/elasticnet problem

Thanks for the suggestions so far, I'll be investigating these options :)

[julia-users] Achitecture for solving largish LASSO/elasticnet problem

Hello I'm trying to solve a largish elasticnet type problem (convex optimisation). - The LARS.jl package produces Out of Memory errors for a test (1000, 262144) problem. /proc/meminfo suggests I have 17x this array size free so not sure what's going on there. - I have access to

[julia-users] Re: Parallel computing: SharedArrays not updating on cluster

As the others have said, it won't work like that. I found a few options: 1. DistributedArrays. Message passing handled in the background. Some limitations, but I've not used much. 2. SharedArrays on each machine. You can share memory between all the pids on a single machine, and

[julia-users] Segfaults in SharedArrays

I had a similar issue. In the end I started using yield(); in code blocks run on remote machines before the actual command. On phone now so hard to look up but amit murphy posted a patch to do with finalisation on remotes that also helped (think issue number might have been 14445).

Re: [julia-users] Abstract version of rng=1:end ?

to be taken using > > dispatch for static optimization. > > > > On Monday, June 13, 2016 at 11:05:47 AM UTC-4, Yichao Yu wrote: > > > On Mon, Jun 13, 2016 at 10:47 AM, Matthew Pearce <mat...@refute.me.uk > > > > > > > wrote:

[julia-users] Abstract version of rng=1:end ?

Hello I find myself frequently wishing to define functions that accept a range argument to operate on like: function foo{T}(M::Array{T,1}, rng::UnitRange) return sum(M[rng].^2) end It would be nice to be able to pass in `rng = 1:end`, but this isn't legal. So I need a signature like,

[julia-users] Re: moving data to workers for distributed workloads

Greg, interesting suggestion about const - I should use that more. In that setup we'd still be transferring alot of big coefficient vectors around (workarounds possible).

[julia-users] Re: moving data to workers for distributed workloads

I might have somewhat oversimplified the example. > > Thanks, > Michael > > On Thursday, May 19, 2016 at 7:41:49 AM UTC-7, Matthew Pearce wrote: >> >> Hi Michael >> >> Your current code looks like will pull back the `coefficients` across the >> network

[julia-users] Re: moving data to workers for distributed workloads

Also... The above doesn't respect ordering of the `myidx` variable. Not sure how your problem domain operates, so it could get more complicated if things like order of execution matter. ;)

[julia-users] Re: moving data to workers for distributed workloads

Hi Michael Your current code looks like will pull back the `coefficients` across the network (500 gb transfer) and as you point out transfer `input` each time. I wrote a package ClusterUtils.jl to handle my own problems (MCMC sampling) which were

[julia-users] Re: GPU capabilities

My university cluster uses Tesla M2090 cards. My experience (not comprehensive) so far is that the CUDArt.jl + CU.jl libraries work as one would expect. They're not all 100% complete and some further documentation in places would be nice, but they're pretty good. The only funny behaviour

[julia-users] Re: Capturing Error messages as strings

ly try writing code implementing any hints / suggestions people might have. On Wednesday, March 30, 2016 at 5:53:53 PM UTC+1, Matthew Pearce wrote: > > > Anyone know how to capture error messages as strings? This is for > debugging of code run on remote machines, as the trace

[julia-users] Capturing Error messages as strings

Anyone know how to capture error messages as strings? This is for debugging of code run on remote machines, as the trace on the master node appears to be incomplete. Note I am *not* asking about the atexit() behaviour. I am asking about capturing non-fatal errors like sqrt(-1) rather than

Re: [julia-users] CUDArt.jl - segfaults after updating Julia

Matthew On Monday, March 21, 2016 at 7:35:40 PM UTC, Stefan Karpinski wrote: > > I'm guessing that CUDArt doesn't support the Julia dev – you're probably > safe using a stable release of Julia instead – i.e. the latest v0.4.x > release. > > On Mon, Mar 21, 2016 at 3:13 PM, M

[julia-users] CUDArt.jl - segfaults after updating Julia

I recently updated Julia to the latest version: ```julia julia> versioninfo() Julia Version 0.5.0-dev+3220 Commit c18bc53 (2016-03-21 11:09 UTC)``` After doing so I get errors when trying to use CUDArt.jl. I'm writing to ask whether my best bet is to fully clean and remake my julia build or

Re: [julia-users] Using CUDArt on remote machines - 'illegal memory access'

Thanks again. Think the problem may have been with my kernel and getting confused about the row major and column major ordering of the layout of the array. I thought I'd checked it was producing the correct norms yesterday, but I must have changed something... To get it straight if A is a

Re: [julia-users] Using CUDArt on remote machines - 'illegal memory access'

Thanks Tim. For me `elty=Float32' so if I use `CudaArray(elty, ones(10))' or `CudaArray( elty, ones(10)...)' I get a conversion error. [I am running Julia 0.5.0-dev+749] The result of my CudaArray creation above looks like: julia> to_host(CudaArray(map(elty, ones(10' 1x10 Array{Float32,2}:

[julia-users] Re: Using CUDArt on remote machines - 'illegal memory access'

I should add that I don't think the error lies in my `reap' function for remote calls. As I can correctly call the cudakernels.sqrownorms function on the host from the remote: julia> reap(3, :(reap(1, :(sum(cudakernels.sqrownorms(d_M[1] ))[3] 5.149127f6 (The above gets process three to

[julia-users] Using CUDArt on remote machines - 'illegal memory access'

Hello I've come across a baffling error. I have a custom CUDA kernel to calculate squared row norms of a matrix. It works fine on the host computer: julia> d_M = residual_shared(Y_init,A_init,S_init,k,sig) CUDArt.CudaArray{Float32,2}(CUDArt.CudaPtr{Float32}(Ptr{Float32} @

[julia-users] Re: Google Summer of Code 2016 - Ideas Page

Whose GSoC idea is the 'Dynamic distributed execution for data parallel tasks in Julia' . I'd be keen to collaborate. I've been working on a package (ClusterUtils.jl

[julia-users] What to do with my package ClusterUtils.jl ?

As part of working on an MCMC algorithm, I have produced some tools to help leverage my academic cluster. I've stuck them on github as ClusterUtils.jl . The functionality builds on `remotecall' and on the ClusterManagers.jl package, with -

[julia-users] Advice on factorisation of sparse banded systems and solvers.

I have a problem which involves sparse, banded, symmetric, PD systems. In particular if A is such a matrix and A=LL' I need to solve L' x = z. The size of A prohibits using full matrices. So I am looking for advice on tactics for this problem, as there doesn't seem to be an out-of-the-box

[julia-users] Re: Cholmod Factor re-use

Thanks, but I'm afraid not. I guess the inclusion of Cholmod into Julia must have been restructured since that part of the code was written. E.g.: ``` julia> using Base.LinAlg.CHOLMOD.CholmodFactor ERROR: UndefVarError: CHOLMOD not defined ``` Matthew On Tuesday, November 24, 2015 at 3:43:21

[julia-users] Cholmod Factor re-use

Hello I was going to investigate whether re-using the structure of a sparse cholesky could save me some time. Trying it, I got this: julia> C = cholfact(K) Base.SparseMatrix.CHOLMOD.Factor{Float64} type: LLt method: supernodal maxnnz: 0 nnz: 4117190 julia> D =

[julia-users] SharedArray - intialisation by filename

So I was poking around the SharedArray code, and saw that there was a method for creating one given a file name. After some further playing about I've realised the file has to be in some sort of binary format (e.g. created by write(somefileio, somearray) ) due to something to do with mmapping

[julia-users] Re: parallel and PyCall

C+10, Yakir Gagnon wrote: >> >> Yea, right? So what’s the answer? How can we if at all do any PyCalls >> parallely? >> >> On Monday, October 26, 2015 at 11:49:35 PM UTC+10, Matthew Pearce wrote: >> >> Thought I had an idea about this, I was wrong: >>&

[julia-users] CUDArt.CudaArry - fill! not working

I'm not having much luck filling a CUDArt.CudaArray matrix with a value. julia> C = CUDArt.CudaArray(Float64, (10,10)) CUDArt.CudaArray{Float64,2}(CUDArt.CudaPtr{Float64}(Ptr{Float64} @ 0x000b034a0e00),(10,10),0) julia> fill!(C, 2.0) ERROR: KeyError: (0,"fill_contiguous",Float64) not found

[julia-users] Re: parallel and PyCall

Thought I had an idea about this, I was wrong: ```julia julia> @everywhere using PyCall julia> @everywhere @pyimport pylab julia> remotecall_fetch(pylab.cumsum, 5, collect(1:10)) ERROR: cannot serialize a pointer [inlined code] from error.jl:21 in serialize at serialize.jl:420 [inlined

[julia-users] Re: CUDArt.CudaArry - fill! not working

Thanks also Tim.

[julia-users] Re: CUDArt.CudaArry - fill! not working

Thanks Kristoffer, indeed that works. I'm slightly baffled by the initialisation process. In my session I had already used fill! successfully on a matrix already on the device, so presumed everything had been initialised. Clearly that's not enough.

[julia-users] Re: Shared Array on remote machines?

Indeed I am not trying to share memory across processes on different machines. Yes neither do I understand why the code above doesn't work. I was hoping I might get suggestions as to what such workaround might be. On Tuesday, October 20, 2015 at 2:10:27 PM UTC+1, Páll Haraldsson wrote: > >

[julia-users] How to send dynamically generated code to worker processes?

Hello I was wondering if there was a way to send dynamically generated code over the network? This is the kind of thing I am imagining to get bar() loaded on worker processes: *workrs = addprocs(4)function foo(x) function bar(y) x*y end barendbar = foo(2)[@spawnat w

Re: [julia-users] How to send dynamically generated code to worker processes?

Thanks, works nicely. On Thursday, October 15, 2015 at 12:39:17 PM UTC+1, Stefan Karpinski wrote: > > Prefix the function definition with @everywhere and it should work. > > On Thu, Oct 15, 2015 at 5:06 PM, Matthew Pearce <mat...@refute.me.uk > > wrote: > >&g

[julia-users] Shared Array on remote machines?

I was wondering if it was possible to spawn shared arrays on remote machines. *julia> S = SharedArray(Int, (3,4), init = S -> S[localindexes(S)] = myid(), pids=Int[1,2])* *3x4 SharedArray{Int64,2}: 1 1 2 2 1 1 2 2 1 1 2 2* *julia> remotecall_fetch(readall, 6,

[julia-users] Re: What's the reason of the Success of Python?

One big pull for python is the ecosystem. Almost any task has a python package available. However, there are gaps. For all the strength of scipy + numpy, there are serious gaps. For instance in methods for sparse matrices, and CUDA bindings. It's those gaps that brought me here. Also, pip is

[julia-users] Re: Pkg.[update()/install()/build()] woes on Windows 10 64 bit

If it's any consolation your line "git config --global url."https://".insteadOf git://" has helped me out of a bind! On Friday, September 25, 2015 at 8:29:14 PM UTC+1, Evan Fields wrote: > > I've been encountering problems with packages. Here's what happened: > >- I installed Julia 0.3.11

[julia-users] Pkg dependence on git://

rrors for the same reason. TL,DR; It would be nice if Pkg did a try except on the url prefix using https://. Cheers Matthew Pearce