Re: [julia-users] Strange ccall behaviour: passing Ptr{Void}s to functions breaks C-API

[Jameson Nash]

I'm not sure if this is directly related, but the result of the syntactic
expression `pointer("y")` is undefined. using that result will cause
undefined behavior. undefined behavior is not bounded to the expression in
question and instead may alter anything it wants about your computer. do
not use the `pointer` function on anything other than the name a variable
(e.g. `pointer(y)`), or better yet, don't use it ever. on Julia 0.4,
`pointer` is simply the unsafe version of `Ref`. and on any version, it is
generally redundant with the automatic conversion implicit in ccall.

also, fwiw, that's not how that c function appears to be defined (
http://opensmt.googlecode.com/svn-history/r30/trunk/src/api/opensmt_c.h),
since it only expects 2 arguments and you are trying to pass 4:

typedef void * opensmt_expr;
typedef void * opensmt_context;

opensmt_expr  opensmt_mk_int_var(opensmt_context, char *);


C actually defines that this does not cause undefined behavior, but it is
also just not very useful here.


On Wed, Apr 15, 2015 at 11:33 PM Zenna Tavares 
wrote:

> I have been wrapping an API for nonlinear constraint solving in Julia0.4
> (commit 176f9bcdc7e745560107038c6fd7333e85bb4c57).
>
> I am getting different behaviour on what should be semantically identical
> programs.  That is, if I pass my a Ptr{Void} into a function my program
> behaves incorrectly.  If I don't use a function, or just use a global
> variable for that Ptr{Void}, it works correctly
>
> The following example uses constraint solving to check whether: if is an
> integer y is between -100 and 100, is there some y such that y < 0.  This
> should of course be satisfiable (e.g. y = -1)
>
> ctx = dReal.opensmt_mk_context(Cuint(1))
>
>
> function test1(ctx)
>   ccall((:opensmt_mk_int_var, "libdreal"), Ptr{Void},
>   (Ptr{Void}, Ptr{UInt8}, Float64, Float64), ctx, pointer("y"), -100,
> 100)
> end
>
> function test2()
>   global ctx
>   ccall((:opensmt_mk_int_var, "libdreal"), Ptr{Void},
>   (Ptr{Void}, Ptr{UInt8}, Float64, Float64), ctx, pointer("y"), -100,
> 100)
> end
>
>
> println("Creating some integer variables")
> y = test1(ctx) # CAUSES INCORRECT RESULT
> #y = test2() # CAUSES CORRECT RESULT
>
> # leq:= y < 0
> zero = dReal.opensmt_mk_num_from_string(ctx, "0" )
> leq1 = dReal.opensmt_mk_leq(ctx, y, zero )
>
>
> # Asserting y < 0
> dReal.opensmt_assert(ctx, leq1)
>
> # Checking for satisfiability
> res = dReal.opensmt_check(ctx)
> @test res == true
>
> Depending on whether I use test1 or test2 I get different results.  How is
> this possible? Is some strange garbage collection effect going on?
>
> Thanks
>

[julia-users] Re: search on docs.julialang.org doesn't seem to work

[Seth]

Yes - see https://github.com/JuliaLang/julia/issues/10831

On Wednesday, April 15, 2015 at 9:05:53 PM UTC-7, Christian Peel wrote:
>
> Searching the documentation on docs.julialang.org doesn't seem to work 
> for me.  I tried in several different browsers, two different macines, and 
> tried it several hours ago and now.I'm certain that I could previously 
> use the 'search docs' box to ... search, but it doesn't seem to work now.  
> Is something broken?
>
> thanks
>
> -- 
> chris...@ieee.org 
>  

[julia-users] search on docs.julialang.org doesn't seem to work

[Christian Peel]

Searching the documentation on docs.julialang.org doesn't seem to work for
me.  I tried in several different browsers, two different macines, and
tried it several hours ago and now.I'm certain that I could previously
use the 'search docs' box to ... search, but it doesn't seem to work now.
Is something broken?

thanks

-- 
chris.p...@ieee.org

[julia-users] Strange ccall behaviour: passing Ptr{Void}s to functions breaks C-API

[Zenna Tavares]

I have been wrapping an API for nonlinear constraint solving in Julia0.4 
(commit 176f9bcdc7e745560107038c6fd7333e85bb4c57).

I am getting different behaviour on what should be semantically identical 
programs.  That is, if I pass my a Ptr{Void} into a function my program 
behaves incorrectly.  If I don't use a function, or just use a global 
variable for that Ptr{Void}, it works correctly

The following example uses constraint solving to check whether: if is an 
integer y is between -100 and 100, is there some y such that y < 0.  This 
should of course be satisfiable (e.g. y = -1)

ctx = dReal.opensmt_mk_context(Cuint(1))


function test1(ctx)
  ccall((:opensmt_mk_int_var, "libdreal"), Ptr{Void},
  (Ptr{Void}, Ptr{UInt8}, Float64, Float64), ctx, pointer("y"), -100, 
100)
end

function test2()
  global ctx
  ccall((:opensmt_mk_int_var, "libdreal"), Ptr{Void},
  (Ptr{Void}, Ptr{UInt8}, Float64, Float64), ctx, pointer("y"), -100, 
100)
end


println("Creating some integer variables")
y = test1(ctx) # CAUSES INCORRECT RESULT
#y = test2() # CAUSES CORRECT RESULT

# leq:= y < 0
zero = dReal.opensmt_mk_num_from_string(ctx, "0" )
leq1 = dReal.opensmt_mk_leq(ctx, y, zero )


# Asserting y < 0
dReal.opensmt_assert(ctx, leq1)

# Checking for satisfiability
res = dReal.opensmt_check(ctx)
@test res == true

Depending on whether I use test1 or test2 I get different results.  How is 
this possible? Is some strange garbage collection effect going on?

Thanks

[julia-users] Re: Need help parallelizing this benchmark

[SixString]

@Kristoffer Carlsson , I do appreciate your help and your clever use of 
Yeppp, which is limited to reals.  I may be able to redesign my algorithm 
with all reals and get faster execution with Julia than with Python, which 
does not have a wrapper for Yeppp that I could find.  Doing so may also 
involve vectorized dot products with the BLAS library.  Since I am 
processing GB-sized vectors, this involves large temporary vectors, and I 
may not have enough RAM.  Also, it contradicts the advice I was given 
previously in this forum to write my loops in pure Julia for speed.  So I 
would still like to hear from forum members about how to get parallelized 
pure Julia code executing faster than single-threaded.  Maybe I have to 
wait for Julia v0.5 for this to manifest.

[julia-users] Re: acoustic simulations in Julia

[Steven G. Johnson]

On Wednesday, April 15, 2015 at 6:55:20 AM UTC-4, Niccolo' Antonello wrote:
>
> I uploaded this to:
> https://github.com/nantonel/RIM.jl 
> 
> and:
> https://github.com/nantonel/Fdtd.jl 
> 
>>
>>
>>
Note that "FDTD" is a bit too generic ... the term could apply to any 
time-dependent PDE, and is commonly used for Maxwell's equations.   I would 
suggest something more specific, like ScalarWaveFDTD

(Wouldn't FDFD, i.e. the frequency-domain scalar Helmholtz equation, be 
more appropriate for this sort of problem?  Depends on how wide a bandwidth 
you care about and what you want to simulate, I guess.)

[julia-users] Re: Need help parallelizing this benchmark

[Kristoffer Carlsson]

Not sure how valuable this post is but for fun I tried the new Yeppp.jl 
wrapper (https://github.com/JuliaLang/Yeppp.jl) and got it down from 15 to 
~ 4 seconds without anything running in parallel. I didn't bother sending 
in the buffer arrays but instead just put them as consts outside the 
function

using Yeppp
const n = 1000
const p = Array(Float64, n)
const p_cos = Array(Float64, n)
const p_sin = Array(Float64, n)


function innerloop(x_real::Array{Float64}, x_imag::Array{Float64},
   y::Array{Complex{Float64}}, scaler::Float64)

   n = length(x_real)
   assert(length(y) == n)

   Yeppp.exp!(p, x_real)
   Yeppp.cos!(p_cos, x_imag)
   Yeppp.sin!(p_sin, x_imag)


   s = 0.0 + 0.0im
  @inbounds for i in 1:n
s += y[i] * Complex(p[i] * scaler * p_cos[i], p[i] * scaler * p_sin[i])
  end
  return s
end


function timeit(n::Int, reps::Int)
   x_real = fill(0.0, n)
   x_imag = fill(-1.0, n)
   y = fill(0.0+1.0im, n)
   res = Array(Complex{Float64}, reps)
   for k in 1:reps
   scaler = convert(Float64, k)
   res[k] = innerloop(x_real, x_imag, y, scaler)
   end
end


timeit(100, 1)


@time timeit(n, 50)




On Wednesday, April 15, 2015 at 8:32:19 PM UTC+2, SixString wrote:
>
> I've been advised to write loops in pure Julia for speed, contrary to the 
> strategy of using library functions in Python.  After spending a couple 
> hundred hours learning Julia, I am having trouble matching Python's speed 
> with Julia.  Here is code for a benchmark in Python:
>
> import numpy as np
>
> import numexpr as ne
>
> from scipy.linalg.blas import zdotu
>
> from timeit import default_timer as timer
>
> n = 1000
>
> reps = 50
>
> x = -1J * np.ones(n)
>
> y = 1J * np.ones(n)
>
> res = np.empty(reps, dtype=np.complex128)
>
> start = timer()
>
> for k in range(reps):
>
>scaler = float(k)
>
>cexp = ne.evaluate('exp(scaler * x)')
>
>res[k] = zdotu(cexp, y)
>
> exec_time=(timer() - start)
>
> print
>
> print("Execution took", str(round(exec_time, 1)), "seconds")
>
> On my Intel Core-i7 Windows PC with 64-bit Python 3.4, this took 4.5 
> seconds to execute.  I note that Python's numexpr library function uses all 
> 8 CPU threads, while BLAS' zdotu() is single-threaded.
>
> Here is my single-threaded Julia code for the same benchmark:
> function innerloop(x::Array{Complex{Float64}}, y::Array{Complex{Float64}}, 
> scaler::Float64)
>
>n = length(x)
>
>assert(length(y) == n)
>
>s = 0.0 + 0.0im
>
>@simd for i in 1:n
>
>@inbounds s += y[i] * exp(scaler * x[i])
>
>end
>
>s
>
> end
>
>
> function timeit(n::Int, reps::Int)
>
>x = fill(0.0-1.0im, n)
>
>y = fill(0.0+1.0im, n)
>
>res = Array(Complex{Float64}, reps)
>
>@time for k in 1:reps
>
>scaler = convert(Float64, k)
>
>res[k] = innerloop(x, y, scaler)
>
>end
>
> end
>
>
> timeit(100, 1)
>
> timeit(1000, 50)
>
> On the same PC with 64-bit Julia 0.3.7, the second call to timeit() took 
> 17.6 seconds and allocated 0 bytes.  That is 3.9 times slower than Python.  
> I note that the @simd decorator didn't speed up the execution.
>
> Python performance seems in reach if I can get a parallelized Julia 
> implementation.  Here is my attempt:
> function innerloop(x::Array{Complex{Float64}}, y::Array{Complex{Float64}}, 
> scaler::Float64)
>
>n = length(x)
>
>assert(length(y) == n)
>
>s = @parallel (+) for i in 1:n
>
>y[i] * exp(scaler * x[i])
>
>end
>
>s
>
> end
>
> The timeit() function remains the same.  Running Julia with the "-p 8" on 
> the command line, the second call to timeit() took 233 seconds and 
> allocated 303GB.  The huge allocation is my clue that something is really 
> wrong here.  All my attempts at adding @inbounds created errors.
>
> I should say that this benchmark is representative of a real problem that 
> I am trying to solve.  My actual Python code takes 10 minutes to execute 
> because n and reps are larger.  However, the types of the vectors match my 
> actual code, and scaler changes with each rep.
>
> Please show me by example how to modify innerloop() to be faster than the 
> single-threaded version.  My preference is for a fix with Julia 0.3.7, but 
> I am also interested if someone provides Julia 0.4-pre code including a 
> speed comparison with the single-threaded version on their machine.  If 
> possible, I would like to avoid SharedArrays because (I believe) they can't 
> be used with BLAS calls in other functions.
>

[julia-users] Re: Julia and Spark

[wildart]

>
> 1) simply wrap the Spark java API via JavaCall. This is the low level 
> approach. BTW I've experimented with javaCall and found it was unstable & 
> also lacking functionality (e.g. there's no way to shutdown the jvm or 
> create a pool of JVM analogous to DB connections) so that might need some 
> work before trying the Spark integration.
>

Using JavaCall is not an option, especially when JVM became close-sourced, 
see https://github.com/aviks/JavaCall.jl/issues/7.

Python bindings are done through Py4J, which is RPC to JVM. If you look at 
the sparkR , it is done in a 
same way. sparkR uses a RPC interface to communicate with a Netty-based 
Spark JVM backend that translates R calls into JVM calls, keeps 
SparkContext on a JVM side, and ships serialized data to/from R.

So it is just a matter of writing Julia RPC to JVM and wrapping necessary 
Spark methods in a Julia friendly way. 

[julia-users] My plugin for integrating Julia inside a Vim notebook

[Thomas Baruchel]

Hi,

I am new to Julia. In order to learn and experiment with it, I first 
checked if wether it was easy or not
to use Julia from my plugin: vim-notebook. It happened to work very well; 
if you are interested by such
a way of working: please, have a look at 
https://github.com/baruchel/vim-notebook
(you may look at the video though other interpreters than Julia are used in 
this video).

Best regards,

tb.

[julia-users] Need help parallelizing this benchmark

[SixString]

I've been advised to write loops in pure Julia for speed, contrary to the 
strategy of using library functions in Python.  After spending a couple 
hundred hours learning Julia, I am having trouble matching Python's speed 
with Julia.  Here is code for a benchmark in Python:

import numpy as np

import numexpr as ne

from scipy.linalg.blas import zdotu

from timeit import default_timer as timer

n = 1000

reps = 50

x = -1J * np.ones(n)

y = 1J * np.ones(n)

res = np.empty(reps, dtype=np.complex128)

start = timer()

for k in range(reps):

   scaler = float(k)

   cexp = ne.evaluate('exp(scaler * x)')

   res[k] = zdotu(cexp, y)

exec_time=(timer() - start)

print

print("Execution took", str(round(exec_time, 1)), "seconds")

On my Intel Core-i7 Windows PC with 64-bit Python 3.4, this took 4.5 
seconds to execute.  I note that Python's numexpr library function uses all 
8 CPU threads, while BLAS' zdotu() is single-threaded.

Here is my single-threaded Julia code for the same benchmark:
function innerloop(x::Array{Complex{Float64}}, y::Array{Complex{Float64}}, 
scaler::Float64)

   n = length(x)

   assert(length(y) == n)

   s = 0.0 + 0.0im

   @simd for i in 1:n

   @inbounds s += y[i] * exp(scaler * x[i])

   end

   s

end


function timeit(n::Int, reps::Int)

   x = fill(0.0-1.0im, n)

   y = fill(0.0+1.0im, n)

   res = Array(Complex{Float64}, reps)

   @time for k in 1:reps

   scaler = convert(Float64, k)

   res[k] = innerloop(x, y, scaler)

   end

end


timeit(100, 1)

timeit(1000, 50)

On the same PC with 64-bit Julia 0.3.7, the second call to timeit() took 
17.6 seconds and allocated 0 bytes.  That is 3.9 times slower than Python.  
I note that the @simd decorator didn't speed up the execution.

Python performance seems in reach if I can get a parallelized Julia 
implementation.  Here is my attempt:
function innerloop(x::Array{Complex{Float64}}, y::Array{Complex{Float64}}, 
scaler::Float64)

   n = length(x)

   assert(length(y) == n)

   s = @parallel (+) for i in 1:n

   y[i] * exp(scaler * x[i])

   end

   s

end

The timeit() function remains the same.  Running Julia with the "-p 8" on 
the command line, the second call to timeit() took 233 seconds and 
allocated 303GB.  The huge allocation is my clue that something is really 
wrong here.  All my attempts at adding @inbounds created errors.

I should say that this benchmark is representative of a real problem that I 
am trying to solve.  My actual Python code takes 10 minutes to execute 
because n and reps are larger.  However, the types of the vectors match my 
actual code, and scaler changes with each rep.

Please show me by example how to modify innerloop() to be faster than the 
single-threaded version.  My preference is for a fix with Julia 0.3.7, but 
I am also interested if someone provides Julia 0.4-pre code including a 
speed comparison with the single-threaded version on their machine.  If 
possible, I would like to avoid SharedArrays because (I believe) they can't 
be used with BLAS calls in other functions.

[julia-users] Re: Best way of using code introduced in 0.4 in 0.3 builds?

[Seth]

On Wednesday, April 15, 2015 at 11:08:27 AM UTC-7, Steven G. Johnson wrote:
>
> On Wednesday, April 15, 2015 at 10:31:46 AM UTC-4, Seth wrote:
>>
>> 1) is getting Pair into 0.3 something that Compat.jl should handle, or 
>> should it be a request to get it backported to the main 0.3 build?
>>
>  
> Maybe it could go into Compat.  But isn't Pair mainly useful when you have 
> the nice a => b syntax for Pair(a,b)?
>
>
I'm using it because it provides a Base-supported "native" type that I can 
use for graph edges (source and dest vertices are the elements of the 
pair). The => construct is just an added bonus. In 0.3 I could default to 
my own custom datatype, but it'd be nice if Pair were available in earlier 
versions. 


2) what's the best way to determine the version number that introduced Pair 
>> in 0.4? (I've tried git bisect but couldn't get a definitive answer.) This 
>> would be the approach I'd take immediately: to have the old type available 
>> if the version of Julia running didn't support Pair.
>>
>
> I would look at git blame on the base/exports.jl file, to see when Pair 
> was exported.  That turns up this commit:
>
>
> https://github.com/JuliaLang/julia/commit/034b8f5599c3dcd3e770ba8af3a9c28ef2497532
>
>  To turn this into a version number, see the little shell script on the 
> Compat.jl page -- it produces 0.4.0-dev+818 for this commit.
>

Thanks. I recall doing something similar a while back but couldn't remember 
the precise steps.
 

[julia-users] Re: Best way of using code introduced in 0.4 in 0.3 builds?

[Steven G. Johnson]

On Wednesday, April 15, 2015 at 10:31:46 AM UTC-4, Seth wrote:
>
> 1) is getting Pair into 0.3 something that Compat.jl should handle, or 
> should it be a request to get it backported to the main 0.3 build?
>
 
Maybe it could go into Compat.  But isn't Pair mainly useful when you have 
the nice a => b syntax for Pair(a,b)?

2) what's the best way to determine the version number that introduced Pair 
> in 0.4? (I've tried git bisect but couldn't get a definitive answer.) This 
> would be the approach I'd take immediately: to have the old type available 
> if the version of Julia running didn't support Pair.
>

I would look at git blame on the base/exports.jl file, to see when Pair was 
exported.  That turns up this commit:

  
 
https://github.com/JuliaLang/julia/commit/034b8f5599c3dcd3e770ba8af3a9c28ef2497532

 To turn this into a version number, see the little shell script on the 
Compat.jl page -- it produces 0.4.0-dev+818 for this commit.

Re: [julia-users] Path of julia executable?

[Isaiah Norton]

You can use JULIA_HOME to get the base path, and the default process_title
should match the binary name. I don't think we save args[0] anywhere.

On Wed, Apr 15, 2015 at 11:05 AM, René Donner  wrote:

> perfect, thanks!
>
> Am 15.04.2015 um 17:02 schrieb Stefan Karpinski :
>
> > Sys.get_process_title()
>
>

Re: [julia-users] Path of julia executable?

[René Donner]

perfect, thanks!

Am 15.04.2015 um 17:02 schrieb Stefan Karpinski :

> Sys.get_process_title()

Re: [julia-users] Path of julia executable?

[Stefan Karpinski]

There's Sys.get_process_title() although that's not necessarily the path.

On Wed, Apr 15, 2015 at 10:58 AM, René Donner  wrote:

> Hi,
>
> is there a way to find out the path of the Julia executable inside of
> Julia, similar to args[0] in C?
>
> Thanks,
>
> Rene
>
>
>
>

[julia-users] Path of julia executable?

[René Donner]

Hi,

is there a way to find out the path of the Julia executable inside of Julia, 
similar to args[0] in C?

Thanks,

Rene

Re: [julia-users] Can't use

[Matt Bauman]

On Wednesday, April 15, 2015 at 10:36:17 AM UTC-4, Marcus Appelros wrote:
>
> The code you posted works fine here... Are you using some other package 
> also that supplies another head? Which versions do you have?
>

Yes, that seems likely.  I imagine you're overwritten (and not extended) 
the `head` name within your current scope — either by `using` another 
package that provides a different head after using DataFrames, or by 
writing your own head methods without explicitly importing DataFrames.head. 
 Perhaps try `DataFrames.head(df)`.

Re: [julia-users] Can't use

[Marcus Appelros]

The code you posted works fine here... Are you using some other package 
also that supplies another head? Which versions do you have?

[julia-users] Best way of using code introduced in 0.4 in 0.3 builds?

[Seth]

Hi,

I've switched over to using Pair in one of my modules. Problem is, it's not 
available in 0.3 (at least not in 0.3.7). Questions:

1) is getting Pair into 0.3 something that Compat.jl should handle, or 
should it be a request to get it backported to the main 0.3 build?

2) what's the best way to determine the version number that introduced Pair 
in 0.4? (I've tried git bisect but couldn't get a definitive answer.) This 
would be the approach I'd take immediately: to have the old type available 
if the version of Julia running didn't support Pair.

3) Am I missing another obvious alternative?

Thanks.

Re: [julia-users] Can't use

[Ivan Ogasawara]

Maybe I did not understand your question, but using DataFrames, I have this:

head(dv::AbstractDataArray{T,1}) at
/home/user/.julia/v0.3/DataArrays/src/datavector.jl:1
head(df::AbstractDataFrame,r::Int64) at
/home/user/.julia/v0.3/DataFrames/src/abstractdataframe/abstractdataframe.jl:141
head(df::AbstractDataFrame) at
/home/user/.julia/v0.3/DataFrames/src/abstractdataframe/abstractdataframe.jl:142


2015-04-15 10:12 GMT-03:00 Marcus Appelros :

> What is the output of methods(head)?

[julia-users] Re: Non-GPL Julia?

[Sebastian Good]

Is producing a non-GPL Julia build still on the radar? It might be a nice 
goal for the 0.4 release, even if we have to build it ourselves (e.g. 
against MKL, etc.)

On Monday, April 21, 2014 at 5:00:47 PM UTC-4, Steven G. Johnson wrote:
>
>
>
> On Monday, April 21, 2014 4:40:38 PM UTC-4, Tobias Knopp wrote:
>>
>> Yes this is awesome work you have done there. Do you plan to implement 
>> the real-data FFT, DCT and DST in pure Julia also? Then one could really 
>> think about moving FFTW into a package. Hopefully its author is ok with 
>> that ;-) 
>>
>
> I plan to implement real-data FFTs, and move FFTW into a package.
>
> Pure-Julia DCT and DST are not in my immediate plans (they are a PITA to 
> do right because there are 16 types, of which 8 are common); my feeling is 
> that the need for these is uncommon enough that it's not terrible to have 
> these in a package instead of in Base.(Hadamard transforms and MDCTs 
> are also currently in packages.)
>

[julia-users] Can't use

[Marcus Appelros]

What is the output of methods(head)?

[julia-users] Re: julia to excel using libxl

[Martin Somers]

I need to be able to write to an Excel sheet and run a macro hence why I 
was looking at running an eternal C library and was interested in 
interfacing directly with it - the mention packages appear to fall short of 
my needs 

it seems I might be able to run nodejs as a master and do my processing 
through julia and then using edge to do what I want in excel -
https://github.com/tjanczuk/edge#scripting-clr-from-nodejs

I was only interested to see if anyone had much success in going the 
external library route with julia and the possible issues I might face 

tks 
M

[julia-users] Re: Julia and Spark

[Steven Sagaert]

I've been comtemplating writing a high level wrapper to Spark myself since 
I'm interested in both Julia & Spark but I was waiting for Julia 0.4 to 
finalize before even starting.
One can do the integration on several levels:
1) simply wrap the Spark java API via JavaCall. This is the low level 
approach. BTW I've experimented with javaCall and found it was unstable & 
also lacking functionality (e.g. there's no way to shutdown the jvm or 
create a pool of JVM analogous to DB connections) so that might need some 
work before trying the Spark integration.
2) Spark 1.3 has now new and high level interfaces: dataframe API for 
accessing data in the form of distributed dataframes & pipeline API to 
compose algo via pipeline framework. By wrapping the spark dataframe with 
julia dataframe you would quickly have a high level (data scientist level) 
interface to Spark. BTW Spark dataframes are actually also FASTER than the 
more low level approaches like java/scala methods calls or Spark SQL 
(intermediate level) because Spark itself can do more optimizations (this 
is similar to how PyData Blaze works). By wrapping the pipeline API one 
could quickly compose Spark algos to create new algos.
3) for an intermediate approach : wrap the Spark SQL API and use SQL to 
query the system.

Personally I would start with dataframe & pipeline API. Maybe later on if 
needed add Spark SQL API and only do the low level stuff last if needed. 
But before interfacing Spark dataframes with julia ones the julia dataframe 
should become more powerful: at least && and || should be allowed in 
indexing for richer "querying" like in R dataframes.

On Wednesday, April 15, 2015 at 11:37:50 AM UTC+2, Tanmay K. Mohapatra 
wrote:
>
> This thread is to discuss Julia - Spark integration further.
>
> This is a continuation of discussions from 
> https://groups.google.com/forum/#!topic/julia-users/LeCnTmOvUbw (the 
> thread topic was misleading and we could not change it).
>
> To summarize briefly, here are a few interesting packages:
> - https://github.com/d9w/Spark.jl
> - https://github.com/jey/Spock.jl
> - https://github.com/benhamner/MachineLearning.jl 
> 
> - packages at https://github.com/JuliaParallel
>
> We can discuss approaches and coordinate efforts towards whichever looks 
> promising.
>

[julia-users] Re: julia to excel using libxl

[Daniel Høegh]

https://github.com/davidanthoff/ExcelReaders.jl is a wraps of the xlrd Python 
package.

[julia-users] Re: acoustic simulations in Julia

[Niccolo' Antonello]

I uploaded this to:
https://github.com/nantonel/RIM.jl 

and:
https://github.com/nantonel/Fdtd.jl 

thanks for pointing that out!
Niccolo'

On Monday, March 16, 2015 at 3:44:52 PM UTC+1, Niccolo' Antonello wrote:
>
> Dear julia fellow users,
>
> I wanted to tell you that I just released some Julia code for acoustic 
> simulations:
> https://github.com/nantonel/JuFdtd
> - a finite difference time domain simulator for room acoustics
> https://github.com/nantonel/JuRIM
> - a image source method room acoustic impulse response generator
> there is lot of space for improvement from the computational point (in 
> particular for the FDTD which is currently not parallelized)
> so feel free to join their development!
>
> Niccolò
>

[julia-users] Re: Symbolic relations package

[Marcus Appelros]

RFC: https://github.com/jhlq/Equations.jl
git://github.com/jhlq/Equations.jl.git

[julia-users] Re: julia to excel using libxl

[Fabian Gans]

I think the package Taro.jl should be able to do this. 

[julia-users] eval/include code without replacing modules

[Tamas Papp]

Hi,

Is there a way to evaluate code that defines modules without replacing
the modules? Eg include'ing a file

--8<---cut here---start->8---
module Foo

type Bar
  baz
end

end
--8<---cut here---end--->8---

this way would be equivalent to

eval(:(type Bar baz end), Foo)

I know that replacing modules is a feature (allows redefinition of types
etc). But sometimes, when developing code interactively, I find that
cumbersome, since that means that instances created previously now have
a different type, etc.

Best,

Tamas

Re: [julia-users] Problem with addprocs

[John]

Hi Rene, that didn't fix the problem, unfortunately.
I did manage to get it to fix it by calling addprocs from a remote machine 
in the cluster.
So I think there is a problem with firewalls -- perhaps the remote machine 
is trying to SSH back into the local one and failing.
It would be nice if these Julia library functions would emit more 
informative error messages or perhaps have a verbose mode.


On Monday, April 13, 2015 at 4:17:07 PM UTC+1, René Donner wrote:
>
> Can you try whether using "tunnel = true" helps? 
>
>   http://docs.julialang.org/en/release-0.3/stdlib/parallel/#Base.addprocs 
>
> When tunnel==false (the default), the workers need to be able to see each 
> other directly, which will not work through firewalls which allow only ssh 
> etc. 
>
>
>
> Am 13.04.2015 um 12:10 schrieb John >: 
>
> > I'm unable to add a remote instance using addprocs (or a machine file). 
> > 
> > This works (without prompt): 
> > ssh xxx.xxx.xxx.xxx 
> > 
> > This command hangs with no output for the first minute: 
> > addprocs(["usern...@xxx.xxx.xxx.xxx"]) 
> > Then after one minute, the following message appears: 
> > Master process (id 1) could not connect within 60.0 seconds 
> > 
> > This command works: 
> > addprocs(["username@mycurrenthostname"]) 
> > 
> > Thanks in advance for your help! 
> > John 
> > 
> > 
> > 
>
>

Re: [julia-users] Re: Spark and Julia

[Tanmay K. Mohapatra]

Hi Jey,

Spock.jl looks interesting. 

Am I right with my understanding that this gives us something like a 
streaming model for calling Julia methods from within Spark?
I am yet to try it and just had a glance at the code, but is a 
JuliaRDD.java file missing there?

It will be great if you can elaborate the interaction between Spark-Julia a 
little more, and probably a simple example.
I have also started a new thread 
here https://groups.google.com/forum/#!topic/julia-users/S_Qcn0tVxJg to 
continue the discussions.

Best,
Tanmay

On Tuesday, April 14, 2015 at 2:35:31 AM UTC+5:30, Jey Kottalam wrote:
>
> Hi julia-users,
>
> I have the beginnings of a Spark interface for Julia up at 
> https://github.com/jey/Spock.jl. This implementation follows the design 
> used in Spark's Python and R bindings. It so far has the core of the Spark 
> RDD interface implemented, but does not yet implement the additional 
> features of spark-core such as broadcast variables and accumulators. Adding 
> interfaces to other Spark components such as spark-mllib would be an 
> additional significant undertaking that should be revisited once the 
> remainder of spark-core has been implemented.
>
> If anyone is interested in working on this, I would be more than happy to 
> provide assistance and guidance.
>
> -Jey
>
>
>
> On Sun, Apr 5, 2015 at 1:59 AM, Viral Shah > 
> wrote:
>
>> It would be nice to co-ordinate these efforts under the JuliaParallel 
>> organization.
>>
>> -viral
>>
>>
>> On Sunday, April 5, 2015 at 9:39:51 AM UTC+5:30, wil...@gmail.com wrote:
>>>
>>> Spark integration is a tricky thing. Python and R bindings go in a great 
>>> length to map language specific functions into Spark JVM library calls. I 
>>> guess same could be done with JavaCall.jl package in a manner similar to 
>>> SparkR. Look at slide 20 from here: http://spark-summit.org/wp-
>>> content/uploads/2014/07/SparkR-SparkSummit.pdf.
>>>
>>> Spark is a clever distributed data access paradigm which grew from 
>>> Hadoop slowness and limitations. I believe that Julia could provide 
>>> competitive model for a distributed data storage given Julia's parallel 
>>> computing approach. Right now, I am writing Julia bindings for Mesos. The 
>>> idea is to provide, though ClusterManager, access to any Mesos-supervised 
>>> distributed system and run Julia code that environment. In conjuction with 
>>> DistributedArrays and DataFrames, it will create powerful toolbox for 
>>> building distributed systems.
>>>
>>> After all, machine learning on JVM, really?!.
>>>
>>> On Saturday, April 4, 2015 at 11:21:35 AM UTC-4, Jeff Waller wrote:



 On Saturday, April 4, 2015 at 2:22:38 AM UTC-4, Viral Shah wrote:
>
> I am changing the subject of this thread from GSOC to Spark. I was 
> just looking around and found this:
>
> https://github.com/d9w/Spark.jl 
> 
>

 Hey, wow, that's interesting, is this an attempt to reimplement Spark 
 or create a binding? 
  

>  
>
 The real question is with all the various systems out there, what is 
> the level of abstraction that julia should work with. Julia's DataFrames 
> is 
> one level of abstraction, which could also transparently map to csv files 
> (rather than doing readtable), or a database table, or an HBase table. 
> Why 
> would Spark users want Julia, and why would Julia users want Spark? I 
> guess 
> if we can nail this down - the rest of the integration is probably easy 
> to 
> figure out.
>
  
 As a potential user, I will try to answer in a few parts

 There are currently 3 official language bindings (Java, Scala, Python) 
 and some unofficial ones as well
 and R in the works; one thing that users would want is whatever the 
 others get but in the language they
 desire with an abstraction similar to the other language bindings so 
 that examples in other languages
 could be readily translated to theirs.

 Whatever the abstraction turns out the be, there are at least 3 big 
 things that Spark offers; simplification,
 speed, and lazy evaluation.  The abstraction should not make that 
 cumbersome.

 For me, the advantage of Julia is the syntax, the speed, and the 
 connection to all of the Julia packages
 and because of that the community of Julia package authors.  The 
 advantage of Spark is the machinery
 of Spark, access to mlib and likewise the community of Spark users.

 How about an example?  This is simply from Spark examples -- good old 
 K-means.  This is assuming
 the Python binding because probably Julia and Python are most alike, 
 how would we expect this to 
 look using Julia?

 from pyspark.mllib.clustering import KMeans
 from numpy import a

[julia-users] Julia and Spark

[Tanmay K. Mohapatra]

This thread is to discuss Julia - Spark integration further.

This is a continuation of discussions 
from https://groups.google.com/forum/#!topic/julia-users/LeCnTmOvUbw (the 
thread topic was misleading and we could not change it).

To summarize briefly, here are a few interesting packages:
- https://github.com/d9w/Spark.jl
- https://github.com/jey/Spock.jl
- https://github.com/benhamner/MachineLearning.jl 

- packages at https://github.com/JuliaParallel

We can discuss approaches and coordinate efforts towards whichever looks 
promising.

[julia-users] julia to excel using libxl

[Martin Somers]

Just wondering if anyone has any insight into using an external library to 
writing julia data to excel  for example 

http://libxl.com/

julia can interface with C function so why not an enitre library any there 
any resources pertaining to such a case - is there a simpler libary other 
than Libxl that can be used

cheers
M

[julia-users] Re: Julia Installation Conflict with R

[Steven Sagaert]

I have both julia 0.3.7 & R v 3.1.2 on the same Ubuntu 14.04.  I first 
installed julia & later on R. The R distrib is however the one from 
Revolution R Open 8.0.2 beta, not the standard one. Seems to work fine.

On Tuesday, April 14, 2015 at 1:37:52 AM UTC+2, Yudong Ma wrote:
>
> Hi.
> I am pretty new to Julia, and I did manage to install Julia on Ubuntu 
> precise 64.
> Everything works except that the installation of Julia updates some 
> libraries and these updates makes the R shared lib /usr/lib/libR.so 
> complain that the symbol _pcre_valid_utf8 is undefined.
>
> The libraries updated by Julia that affect R are libpcre3, libpcrecpp0, 
> libpcre3-dev.
>
> I am wondering have any of julia users have encounted this issue, and how 
> should I resolve this issue?
> Best Regards
>
>

[julia-users] Re: acoustic simulations in Julia

[Niccolo' Antonello]

I uploaded this to:
https://github.com/nantonel/JuRIM.jl
and:
https://github.com/nantonel/JuFdtd.jl
thanks for pointing that out!
Niccolo'

On Monday, March 16, 2015 at 3:44:52 PM UTC+1, Niccolo' Antonello wrote:
>
> Dear julia fellow users,
>
> I wanted to tell you that I just released some Julia code for acoustic 
> simulations:
> https://github.com/nantonel/JuFdtd
> - a finite difference time domain simulator for room acoustics
> https://github.com/nantonel/JuRIM
> - a image source method room acoustic impulse response generator
> there is lot of space for improvement from the computational point (in 
> particular for the FDTD which is currently not parallelized)
> so feel free to join their development!
>
> Niccolò
>