Re: [julia-users] [Parallel] Using shared memory + parallel maps elegantly

[Amit Murthy]

I have not gone through your post in detail, but would like to point out
that SharedArray can only be used for bitstypes.


On Wed, Jan 22, 2014 at 12:23 PM, Madeleine Udell  wrote:

> # Say I have a list of tasks, eg tasks i=1:n
> # For each task I want to call a function foo
> # that depends on that task and some fixed data
> # I have many types of fixed data: eg, arrays, dictionaries, integers, etc
>
> # Imagine the data comes from eg loading a file based on user input,
> # so we can't hard code the data into the function foo
> # although it's constant during program execution
>
> # If I were doing this in serial, I'd do the following
>
> type MyData
> myint
> mydict
> myarray
> end
>
> function foo(task,data::MyData)
> data.myint + data.myarray[data.mydict[task]]
> end
>
> n = 10
> const data = MyData(rand(),Dict(1:n,randperm(n)),randperm(n))
>
> results = zeros(n)
> for i = 1:n
> results[i] = foo(i,data)
> end
>
> # What's the right way to do this in parallel? Here are a number of ideas
> # To use @parallel or pmap, we have to first copy all the code and data
> everywhere
> # I'd like to avoid that, since the data is huge (10 - 100 GB)
>
> @everywhere begin
> type MyData
> myint
> mydict
> myarray
> end
>
> function foo(task,data::MyData)
> data.myint + data.myarray[data.mydict[task]]
> end
>
> n = 10
> const data = MyData(rand(),Dict(1:n,randperm(n)),randperm(n))
> end
>
> ## @parallel
> results = zeros(n)
> @parallel for i = 1:n
> results[i] = foo(i,data)
> end
>
> ## pmap
> @everywhere foo(task) = foo(task,data)
> results = pmap(foo,1:n)
>
> # To avoid copying data, I can make myarray a shared array
> # In that case, I don't want to use @everywhere to put data on each
> processor
> # since that would reinstantiate the shared array.
> # My current solution is to rewrite my data structure to *not* include
> myarray,
> # and pass the array to the function foo separately.
> # But the code gets much less pretty as I tear apart my data structure,
> # especially if I have a large number of shared arrays.
> # Is there a way for me to avoid this while using shared memory?
> # really, I'd like to be able to define my own shared memory data types...
>
> @everywhere begin
> type MySmallerData
> myint
> mydict
> end
>
> function foo(task,data::MySmallerData,myarray::SharedArray)
> data.myint + myarray[data.mydict[task]]
> end
>
> n = 10
> const data = MySmallerData(rand(),Dict(1:n,randperm(n)))
> end
>
> myarray = SharedArray(randperm(n))
>
> ## @parallel
> results = zeros(n)
> @parallel for i = 1:n
> results[i] = foo(i,data,myarray)
> end
>
> ## pmap
> @everywhere foo(task) = foo(task,data,myarray)
> results = pmap(foo,1:n)
>
> # Finally, what can I do to avoid copying mydict to each processor?
> # Is there a way to use shared memory for it?
> # Once again, I'd really like to be able to define my own shared memory
> data types...
>

[julia-users] [Parallel] Using shared memory + parallel maps elegantly

[Madeleine Udell]

# Say I have a list of tasks, eg tasks i=1:n
# For each task I want to call a function foo
# that depends on that task and some fixed data
# I have many types of fixed data: eg, arrays, dictionaries, integers, etc

# Imagine the data comes from eg loading a file based on user input,
# so we can't hard code the data into the function foo 
# although it's constant during program execution

# If I were doing this in serial, I'd do the following

type MyData
myint
mydict
myarray
end

function foo(task,data::MyData)
data.myint + data.myarray[data.mydict[task]]
end

n = 10
const data = MyData(rand(),Dict(1:n,randperm(n)),randperm(n))

results = zeros(n)
for i = 1:n
results[i] = foo(i,data)
end

# What's the right way to do this in parallel? Here are a number of ideas
# To use @parallel or pmap, we have to first copy all the code and data 
everywhere
# I'd like to avoid that, since the data is huge (10 - 100 GB)

@everywhere begin
type MyData
myint
mydict
myarray
end

function foo(task,data::MyData)
data.myint + data.myarray[data.mydict[task]]
end

n = 10
const data = MyData(rand(),Dict(1:n,randperm(n)),randperm(n))
end

## @parallel
results = zeros(n)
@parallel for i = 1:n
results[i] = foo(i,data)
end

## pmap
@everywhere foo(task) = foo(task,data)
results = pmap(foo,1:n)

# To avoid copying data, I can make myarray a shared array
# In that case, I don't want to use @everywhere to put data on each 
processor
# since that would reinstantiate the shared array.
# My current solution is to rewrite my data structure to *not* include 
myarray,
# and pass the array to the function foo separately.
# But the code gets much less pretty as I tear apart my data structure,
# especially if I have a large number of shared arrays. 
# Is there a way for me to avoid this while using shared memory?
# really, I'd like to be able to define my own shared memory data types...

@everywhere begin
type MySmallerData
myint
mydict
end

function foo(task,data::MySmallerData,myarray::SharedArray)
data.myint + myarray[data.mydict[task]]
end

n = 10
const data = MySmallerData(rand(),Dict(1:n,randperm(n)))
end

myarray = SharedArray(randperm(n))

## @parallel
results = zeros(n)
@parallel for i = 1:n
results[i] = foo(i,data,myarray)
end

## pmap
@everywhere foo(task) = foo(task,data,myarray)
results = pmap(foo,1:n)

# Finally, what can I do to avoid copying mydict to each processor?
# Is there a way to use shared memory for it?
# Once again, I'd really like to be able to define my own shared memory 
data types...

Re: [julia-users] Q: tutorial for working with Julla singleton types

[Ivar Nesje]

Just remember to declare all the variables in the module as const (not just the 
first :D )

Re: [julia-users] [Parallel] Chaining dependencies on modules

[Amit Murthy]

amitm:/tmp$ julia -p 2 -e "@everywhere include(\"module.jl\")"
Warning: requiring "DataStructures" did not define a corresponding module.
Warning: requiring "DataStructures" did not define a corresponding module.

amitm:/tmp$ julia -p 2 -e "require(\"module.jl\")"

Only the include is throwing up the warning, require seems to be fine




On Wed, Jan 22, 2014 at 11:41 AM, Amit Murthy  wrote:

> I just did a 'Pkg.add("DataStructures")' and tried the above code. Seeing
> the same issue.
>
>
> On Wed, Jan 22, 2014 at 11:38 AM, Jeff Bezanson 
> wrote:
>
>> This ought to work. The warning is interesting, since the
>> DataStructures package does (for me at least) define a DataStructures
>> module. Is it possible DataStructures is not fully installed, missing
>> files or something like that?
>>
>>
>> On Wed, Jan 22, 2014 at 1:01 AM, Madeleine Udell
>>  wrote:
>> > I'm trying to understand the most Julian way to perform a particular
>> > parallel programming task. Suppose I need function foo from module.jl
>> to be
>> > available everywhere. Let's call the following code map_foo.jl:
>> >
>> > @everywhere include("module.jl")
>> > @everywhere using MyModule
>> > pmap(foo,1:100)
>> >
>> > That works fine, except when module.jl itself has other dependencies on
>> > other modules:
>> >
>> > module MyModule
>> >
>> > using DataStructures
>> > export foo
>> >
>> > function foo(i)
>> > return Queue(i)
>> > end
>> >
>> > end # module
>> >
>> > In this case, it works to call
>> >
>> > julia map_foo.jl
>> >
>> > but when I call
>> >
>> > julia -p 2 map_foo.jl
>> >
>> > I get the following error
>> >
>> > Warning: requiring "DataStructures" did not define a corresponding
>> module.
>> > Warning: requiring "DataStructures" did not define a corresponding
>> module.
>> > exception on exception on 2: 3: ERROR: ERROR: Queue not definedQueue not
>> > defined
>> >  in
>> >  in foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
>> > (1)/src/julia/questions/module.jl:7
>> >  in anonymous at multi.jl:834
>> >  in run_work_thunk at multi.jl:575
>> >  in anonymous at task.jl:834
>> > foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
>> > (1)/src/julia/questions/module.jl:7
>> >  in anonymous at multi.jl:834
>> >  in run_work_thunk at multi.jl:575
>> >  in anonymous at task.jl:834
>> >
>> > Does anyone know how I can successfully chain dependencies like this
>> when
>> > using parallelism? Calling @everywhere on the import call in module.jl
>> also
>> > doesn't fix the problem, strangely enough.
>> >
>> > Of course, if I could put all my code into shared memory, I'd be much
>> > happier. I just saw an update adding shared memory arrays, but I don't
>> know
>> > if there's a way to get shared memory code!
>> >
>>
>
>

Re: [julia-users] [Parallel] Chaining dependencies on modules

[Amit Murthy]

I just did a 'Pkg.add("DataStructures")' and tried the above code. Seeing
the same issue.


On Wed, Jan 22, 2014 at 11:38 AM, Jeff Bezanson wrote:

> This ought to work. The warning is interesting, since the
> DataStructures package does (for me at least) define a DataStructures
> module. Is it possible DataStructures is not fully installed, missing
> files or something like that?
>
>
> On Wed, Jan 22, 2014 at 1:01 AM, Madeleine Udell
>  wrote:
> > I'm trying to understand the most Julian way to perform a particular
> > parallel programming task. Suppose I need function foo from module.jl to
> be
> > available everywhere. Let's call the following code map_foo.jl:
> >
> > @everywhere include("module.jl")
> > @everywhere using MyModule
> > pmap(foo,1:100)
> >
> > That works fine, except when module.jl itself has other dependencies on
> > other modules:
> >
> > module MyModule
> >
> > using DataStructures
> > export foo
> >
> > function foo(i)
> > return Queue(i)
> > end
> >
> > end # module
> >
> > In this case, it works to call
> >
> > julia map_foo.jl
> >
> > but when I call
> >
> > julia -p 2 map_foo.jl
> >
> > I get the following error
> >
> > Warning: requiring "DataStructures" did not define a corresponding
> module.
> > Warning: requiring "DataStructures" did not define a corresponding
> module.
> > exception on exception on 2: 3: ERROR: ERROR: Queue not definedQueue not
> > defined
> >  in
> >  in foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
> > (1)/src/julia/questions/module.jl:7
> >  in anonymous at multi.jl:834
> >  in run_work_thunk at multi.jl:575
> >  in anonymous at task.jl:834
> > foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
> > (1)/src/julia/questions/module.jl:7
> >  in anonymous at multi.jl:834
> >  in run_work_thunk at multi.jl:575
> >  in anonymous at task.jl:834
> >
> > Does anyone know how I can successfully chain dependencies like this when
> > using parallelism? Calling @everywhere on the import call in module.jl
> also
> > doesn't fix the problem, strangely enough.
> >
> > Of course, if I could put all my code into shared memory, I'd be much
> > happier. I just saw an update adding shared memory arrays, but I don't
> know
> > if there's a way to get shared memory code!
> >
>

Re: [julia-users] [Parallel] Chaining dependencies on modules

[Jeff Bezanson]

This ought to work. The warning is interesting, since the
DataStructures package does (for me at least) define a DataStructures
module. Is it possible DataStructures is not fully installed, missing
files or something like that?


On Wed, Jan 22, 2014 at 1:01 AM, Madeleine Udell
 wrote:
> I'm trying to understand the most Julian way to perform a particular
> parallel programming task. Suppose I need function foo from module.jl to be
> available everywhere. Let's call the following code map_foo.jl:
>
> @everywhere include("module.jl")
> @everywhere using MyModule
> pmap(foo,1:100)
>
> That works fine, except when module.jl itself has other dependencies on
> other modules:
>
> module MyModule
>
> using DataStructures
> export foo
>
> function foo(i)
> return Queue(i)
> end
>
> end # module
>
> In this case, it works to call
>
> julia map_foo.jl
>
> but when I call
>
> julia -p 2 map_foo.jl
>
> I get the following error
>
> Warning: requiring "DataStructures" did not define a corresponding module.
> Warning: requiring "DataStructures" did not define a corresponding module.
> exception on exception on 2: 3: ERROR: ERROR: Queue not definedQueue not
> defined
>  in
>  in foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
> (1)/src/julia/questions/module.jl:7
>  in anonymous at multi.jl:834
>  in run_work_thunk at multi.jl:575
>  in anonymous at task.jl:834
> foo at /Users/madeleineudell/Dropbox/pestilli_icme_life
> (1)/src/julia/questions/module.jl:7
>  in anonymous at multi.jl:834
>  in run_work_thunk at multi.jl:575
>  in anonymous at task.jl:834
>
> Does anyone know how I can successfully chain dependencies like this when
> using parallelism? Calling @everywhere on the import call in module.jl also
> doesn't fix the problem, strangely enough.
>
> Of course, if I could put all my code into shared memory, I'd be much
> happier. I just saw an update adding shared memory arrays, but I don't know
> if there's a way to get shared memory code!
>

[julia-users] [Parallel] Chaining dependencies on modules

[Madeleine Udell]

I'm trying to understand the most Julian way to perform a particular 
parallel programming task. Suppose I need function foo from module.jl to be 
available everywhere. Let's call the following code map_foo.jl:

@everywhere include("module.jl")
@everywhere using MyModule
pmap(foo,1:100)

That works fine, except when module.jl itself has other dependencies on 
other modules:

module MyModule

using DataStructures
export foo

function foo(i)
return Queue(i)
end

end # module

In this case, it works to call 

julia map_foo.jl

but when I call

julia -p 2 map_foo.jl

I get the following error

Warning: requiring "DataStructures" did not define a corresponding module.
Warning: requiring "DataStructures" did not define a corresponding module.
exception on exception on 2: 3: ERROR: ERROR: Queue not definedQueue not 
defined
 in
 in foo at /Users/madeleineudell/Dropbox/pestilli_icme_life 
(1)/src/julia/questions/module.jl:7
 in anonymous at multi.jl:834
 in run_work_thunk at multi.jl:575
 in anonymous at task.jl:834
foo at /Users/madeleineudell/Dropbox/pestilli_icme_life 
(1)/src/julia/questions/module.jl:7
 in anonymous at multi.jl:834
 in run_work_thunk at multi.jl:575
 in anonymous at task.jl:834

Does anyone know how I can successfully chain dependencies like this when 
using parallelism? Calling @everywhere on the import call in module.jl also 
doesn't fix the problem, strangely enough.

Of course, if I could put all my code into shared memory, I'd be much 
happier. I just saw an update  
adding 
shared memory arrays, but I don't know if there's a way to get shared 
memory code!

Re: [julia-users] Q: tutorial for working with Julla singleton types

[Alexander Samoilov]

Thanks, good point,

Summing up, this idiom is well suited for keeping constants, say

module MathConstants
  export pi, e, speed_of_light
  const pi = 3.14159265897
  e = 2.718281828
  speed_of_light = 3e9
...
end

after adding

using MathConstants

it is possible to use pi without prefix (actually pi is already defined, 
this is just for example).

For more general use, say reading parameters from config files (i.e. values 
should mutate), it is better to stick to what Stefan proposed.

Thanks for the comments,
Alexander


On Tuesday, January 21, 2014 10:05:10 PM UTC+4, Ivar Nesje wrote:
>
> Another possibility is to use a module with declared constants.
>
> module CParam
>export rho, cc # the exports are optional but allows you to type "using 
> CParam" and use rho and cc without CParam prefix
>
>const rho = 3.2
>const cc = 54.4
> end
>
> Or you can just use rho and cc as global constants.
>
> Just one word of caution. If you want performance you need to remember the 
> "const" declaration. Currently the julia runtime system assumes that 
> globals might change their type at any point, so it will have to check the 
> type every time it reads the variable.
>
> Ivar
>
> kl. 17:42:18 UTC+1 tirsdag 21. januar 2014 skrev Alexander Samoilov 
> følgende:
>>
>> Ok, thanks.
>>
>> I got the idea.
>>
>> On Tuesday, January 21, 2014 8:31:19 PM UTC+4, Stefan Karpinski wrote:
>>>
>>> Julia's singleton types are abstract types whose only instance is a 
>>> single type object. This is useful for writing methods that dispatch on 
>>> type values rather than on the type of an argument. They are abstract types 
>>> – they don't have fields and they don't hold data.
>>>
>>> If you want an object to store some values, define a type and make a 
>>> single instance of it:
>>>
>>> type CParam
>>>   rho::Float64
>>>   cc::Float64
>>> end
>>> const cparam = CParam(0,0)
>>>
>>>
>>> Now use cparam as your "singleton" – if you don't want more than one 
>>> instance, don't create more than one.
>>>
>>>
>>>
>>> On Tue, Jan 21, 2014 at 10:57 AM, Alexander Samoilov <
>>> alexander...@googlemail.com> wrote:
>>>
 Hello Experts!

 After reading Julia docs on types it is still unclear for me how how to 
 use singleton classes for common tasks, e.g. for packaging some common 
 values.

 As a side note let me elaborate with the example from Scala programming 
 language.
 Scala has intrinsic support for singletons - Objects.

 ```scala
 scala> object CParam {
  | var rho: Double = 1.0
  | var cc: Double = 1.0
  | }
 defined module CParam

 scala> CParam.rho = 2.0
 CParam.rho: Double = 2.0

 scala> import CParam._
 import CParam._

 scala> rho
 res0: Double = 2.0
 ```

 Now trying to project it to Julia:

 ```jlcon
 julia> type CParam
  rho::Float64
  cc::Float64
  function CParam()
new(0.0, 0.0)
  end
end

 julia> x = CParam()
 CParam(0.0,0.0)

 julia> x.rho
 0.0
 ```

 So good so far, though this is not a singleton, just an instance.
 The Julia documentation says that parametric Type{T} is a special 
 parametric kind of the type T - the singleton type.


 ```jlcon
 julia> x = Type{CParam}
 Type{CParam}

 julia> x.rho
 ERROR: type DataType has no field rho
 ```

 Trying to exploit modules as model for singleton class (follow 
 Fortran90 - it uses modules for information hiding and packaging).

 ```jlcon
 julia> module CParam
 export rho, bulk, cc, zz
 rho  = 1.0 
 bulk = 1.0 
 cc   = 1.0 
 zz   = 1.0 
end
  
 julia> CParam.rho  = 2.0
 ERROR: cannot assign variables in other modules
 ```
 Could you please recommend a programming idiom better fitting to Julia?

 Thanks!
 Alexander


>>>

Re: [julia-users] Re: OSX and Linux Sublime-Julia testers?

[Jacob Quinn]

If anyone is interested, I've just pushed a load of changes to the 
Sublime-IJulia project: https://github.com/karbarcca/Sublime-IJulia

I'd love to hear if people are able to get it installed easily on OSX/Linux 
by following the new documentation on the README in the link above. If 
there are no troubles, I'll announce it officially in the groups.

-Jacob


On Tuesday, December 3, 2013 3:27:26 PM UTC-5, Jacob Quinn wrote:
>
> Ctrl+shift+p, type "Install IJulia", then you can open a terminal by 
> Ctrl+shift+p, "Open IJulia". I think I mentioned it in this same thread in 
> more detail is that helps.
>
> -Jacob
>
>
> On Tue, Dec 3, 2013 at 12:18 PM, Diego Javier Zea 
> 
> > wrote:
>
>> How can I istall it on Ubuntu 13.10 ?
>> I have IJulia and Sublime Text 3
>> Best
>>
>
>

Re: [julia-users] Unexpected socket close?

[Amit Murthy]

I can't recall anything that has changed in the parallel codebase recently.
You could try with the 0.2 version just to be sure. Maybe the the julia
processes on the cluster are dying soon after they launch, and hence the
closed connection while reading port information? Could you try launching
julia manually on any of the nodes of the cluster, just to ensure that the
julia setup on those nodes are OK?


On Wed, Jan 22, 2014 at 5:07 AM, David Bindel wrote:

>
>
> I wrote a cluster manager for launching jobs on 
> HTCondora little while back, 
> and was having good luck with it, but now I seem to be
> having some trouble.  The basic logic is that Julia starts a TCP server and
> launches jobs on the cluster that then connect to the server and send back
> their information (by piping through telnet).  The problem is that
> somewhere between when the connection is accepted and when Julia tries to
> read the port information, "the connection is closed by the foreign host".
>
> I've rebuilt Julia between when I last tested this and now, so it's
> possible that there was some change in Julia; it's also possible that there
> was a change in the cluster configuration, since that's equally a moving
> target.  But I'm a bit foxed, and any insights would be welcome.
>
> Cheers,
> David
>

Re: [julia-users] Higher order derivatives in Calculus

[John Myles White]

If you’re willing to wait, I’m happy to return to the Calculus package in the 
spring. I’m focusing on DataFrames/DataArrays (and some database stuff that’s 
closely related) until then.

 — John

On Jan 21, 2014, at 8:42 AM, Hans W Borchers  wrote:

> Thanks for these encouraging words. I have already written an R package with 
> more than a hundred numerical functions (incl. several numerical 
> derivatives), and I would be willing to help build up a numerical package in 
> Julia. But of course, someone from the Julia community will be needed to take 
> the lead. Please let me know when this 'management position'(?) has been 
> taken.
> 
> On Tuesday, January 21, 2014 4:44:37 PM UTC+1, John Myles White wrote:
> Just to chime in: the biggest problem with the Calculus isn’t the absence of 
> usable functionality, it’s that the published interface isn’t a very good one 
> and the more reliable interface, including things like 
> finite_difference_hessian, isn’t exported. 
> 
> To fix this, we need someone to come in and do some serious design work, 
> where they'll rethink interfaces and remove out-dated functionality. As Tim 
> Holy mentioned, the combination of the unpublished finite diference methods 
> and automatic differentation methods in DualNumbers should get you very far. 
> 
>  — John 
> 

Re: [julia-users] Re: Multiple plots on one canvas - IJulia / PyPlot

[Adrian Cuthbertson]

Regarding the Chrome problem, after ensuring I had the latest updates and
restarting the browser, the problem disappeared. Chrome worked equivalently
to FF.


On Tue, Jan 21, 2014 at 10:59 PM, Steven G. Johnson
wrote:

> I'm getting a little frustrated with these SVG issues.  There is also the
> problem that, for plots with zillions of data points, SVG display is
> frighteningly slow.  I'm thinking of just turning off SVG output in PyPlot,
> by default, with a runtime option to re-enable it.
>

Re: [julia-users] Julia computational efficiency vs C vs Java vs Python vs Cython

[Ivan Tong]

I think the greedy solution is both behaviors? =D

Is it possible to give the option to the user?  Maybe a keyword argument 
for each variable or some other notation?  Example

a::Int32
b::_Int32_

so a & b are both Int32, but one allows promotion, the other doesn't?  (or 
some other symbol/keyword...)  Just a greedy suggestion...

(and some sort of rule when you do a+b?  idk haha)

I'm writing a Cython reader for a custom file converter (and wishing Julia 
was more mature, since I'm a long time Matlab user).  I need a lot of 
bit-masking and using pointers with a lot of explicit casting to the input 
mmap file.  So for me personally, right now, I think I'd want or even need 
to have variables be the type I specify.  (Example, I've been accessing 
individual bytes instead of using shifts whenever possible, because it 
seemed a bit faster when I benchmarked it.  On a Big Endian arch, promotion 
might give me hard to track down errors?  Maybe?).

On Thursday, January 16, 2014 8:36:18 AM UTC-5, Stefan Karpinski wrote:
>
> On Thu, Jan 16, 2014 at 4:43 AM, Milan Bouchet-Valat 
> 
> > wrote:
>
>> Don't be sorry -- they only get what they deserve for creating Julia: we 
>> have become greedy too. ;-)
>
>
> I did literally ask for it.
>

Re: [julia-users] Why isn't typeof(Float64[]) <: typeof(Real[]) true?

[Kevin Squire]

Basically because the size of Float64 is known, Float64[] is stored
compactly in memory and allows the use of optimized mathematical routines.
A Real Array can hold any type of real simultaneously (Float64, Float32,
BigFloat), and therefore is stored as an array of pointers.

The general way to define behaviors which work with all Real arrays is to
parametrize functions based on Array element type:

function myfunc{T<:Real}(x::Array{T})
   # ... do something fancy
end

See the following manual sections for more details:

http://docs.julialang.org/en/latest/manual/types/#man-parametric-types
http://docs.julialang.org/en/latest/manual/methods/#parametric-methods

Cheers!

   Kevin

On Tue, Jan 21, 2014 at 4:17 PM, Patrick Foley  wrote:

> Is there a way to get around this?  I have a lot of types (foo1, foo2,
> ) all of which are subtypes of an abstract (bar).  I want to be able to
> define the behavior for arrays of any of the foos just by defining the
> behavior of an array of 'bar's.  Any advice?
>

[julia-users] Why isn't typeof(Float64[]) <: typeof(Real[]) true?

[Patrick Foley]

Is there a way to get around this?  I have a lot of types (foo1, foo2, 
) all of which are subtypes of an abstract (bar).  I want to be able to 
define the behavior for arrays of any of the foos just by defining the 
behavior of an array of 'bar's.  Any advice?

[julia-users] Unexpected socket close?

[David Bindel]

I wrote a cluster manager for launching jobs on 
HTCondora little while back, and 
was having good luck with it, but now I seem to be 
having some trouble.  The basic logic is that Julia starts a TCP server and 
launches jobs on the cluster that then connect to the server and send back 
their information (by piping through telnet).  The problem is that 
somewhere between when the connection is accepted and when Julia tries to 
read the port information, "the connection is closed by the foreign host".

I've rebuilt Julia between when I last tested this and now, so it's 
possible that there was some change in Julia; it's also possible that there 
was a change in the cluster configuration, since that's equally a moving 
target.  But I'm a bit foxed, and any insights would be welcome.

Cheers,
David

[julia-users] Re: ERROR: no method getindex(Array{Any,1},ASCIIString)

[Kees van Prooijen]

We can't repro this on OSX either.
Did you try Stefan's suggestion, enter even simpler one-liners in the 
editor. 


On Tuesday, January 21, 2014 6:56:34 AM UTC-8, Ben Simmons wrote:
>
> I've just installed Julia Studio last night to try and do some work on 
> cellular automata.
>
> I've been familiarising myself with the language, and so far it's worked 
> fine. But when I try to use the if command, I get the following error:
>
> ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
>  
> in Message at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
>  
> in process_incoming at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
>  
> in anonymous at multi.jl:110
>
> This is when running even very simple code, such as:
>
> x=2
> y=4
> if x < y
>   println("x is less than y")
> elseif x > y
>   println("x is greater than y")
> else
>   println("x is equal to y")
> end
>
> Or
>
> x=2
> if x == 2
>   println("x is 2")
> end
>
> Any ideas? I also can't get it to work in Julia (rather than Julia Studio)
>

Re: [julia-users] Re: Multiple plots on one canvas - IJulia / PyPlot

[Steven G. Johnson]

I'm getting a little frustrated with these SVG issues.  There is also the 
problem that, for plots with zillions of data points, SVG display is 
frighteningly slow.  I'm thinking of just turning off SVG output in PyPlot, 
by default, with a runtime option to re-enable it.

[julia-users] Parsing Julia Forms

[Jake Bolewski]

I was playing around with getting LightTable to talk to Julia and one issue 
I ran into was that it was difficult for me to reliably get the source line 
for the end of a Julia form (correctly getting the starting point was 
relatively easy).  Has anyone run into this and come up with a solution?

Best,
Jake

Re: [julia-users] Type Conversion and Promotion for 'in' operator

[Stefan Karpinski]

This is very similar to how we used to do hashing. It would be fine if there 
were a fixed collection of numeric types in Julia, but if course that's not the 
case and user-defined types need to be able to participate in the hashing 
behavior, which rapidly spirals out of control. That's what motivated the 
change to the current behavior, which unfortunately leaves a rather large gap 
in functionality since there's no good way to express equality comparison that 
doesn't care about type but considers NaNs to be equal values – which happens 
to be what I think hashing should probably do.

> On Jan 21, 2014, at 12:53 PM, Sharmila Gopirajan 
>  wrote:
> 
> Thanks for the heads up.  I will use the master then. I am still interested 
> in implementing the hashing strategy for numbers.  So any feedback would be 
> great.
> 
> Regards,
> Sharmi
> 
> 
>> On Tue, Jan 21, 2014 at 10:53 PM, Milan Bouchet-Valat  
>> wrote:
>>> Le mardi 21 janvier 2014 à 00:13 -0500, Jeff Bezanson a écrit :
>>> The main reason is that there are many types of numbers, with more
>>> added all the time. And for purposes of hash tables, it is difficult
>>> to ensure that all numerically-equal numbers hash the same. So we had
>>> isequal(), which is used by dictionaries, distinguish numbers of
>>> different types. At this point, we would kind of like to change this
>>> back and make isequal more liberal (although it would still
>>> distinguish -0.0 and 0.0, and so not be strictly more liberal than
>>> ==). However, the hashing problem remains. Any ideas are welcome.
>> Actually, you changed the behavior of in to use == instead of isequal() 
>> after I filed an issue: https://github.com/JuliaLang/julia/issues/4941
>> 
>> 
>> With git master as of a few days, this works:
>> 
>> julia> x = int32(4)
>> 4
>> 
>> julia> y = int64(4)
>> 4
>> 
>> julia> x == y
>> true
>> 
>> julia> x in [y]
>> true
>> 
>> That doesn't mean hashing shouldn't be improved, though.
>> 
>> 
>> Regards
> 

Re: [julia-users] Q: tutorial for working with Julla singleton types

[Ivar Nesje]

Another possibility is to use a module with declared constants.

module CParam
   export rho, cc # the exports are optional but allows you to type "using 
CParam" and use rho and cc without CParam prefix

   const rho = 3.2
   const cc = 54.4
end

Or you can just use rho and cc as global constants.

Just one word of caution. If you want performance you need to remember the 
"const" declaration. Currently the julia runtime system assumes that 
globals might change their type at any point, so it will have to check the 
type every time it reads the variable.

Ivar

kl. 17:42:18 UTC+1 tirsdag 21. januar 2014 skrev Alexander Samoilov 
følgende:
>
> Ok, thanks.
>
> I got the idea.
>
> On Tuesday, January 21, 2014 8:31:19 PM UTC+4, Stefan Karpinski wrote:
>>
>> Julia's singleton types are abstract types whose only instance is a 
>> single type object. This is useful for writing methods that dispatch on 
>> type values rather than on the type of an argument. They are abstract types 
>> – they don't have fields and they don't hold data.
>>
>> If you want an object to store some values, define a type and make a 
>> single instance of it:
>>
>> type CParam
>>   rho::Float64
>>   cc::Float64
>> end
>> const cparam = CParam(0,0)
>>
>>
>> Now use cparam as your "singleton" – if you don't want more than one 
>> instance, don't create more than one.
>>
>>
>>
>> On Tue, Jan 21, 2014 at 10:57 AM, Alexander Samoilov <
>> alexander...@googlemail.com> wrote:
>>
>>> Hello Experts!
>>>
>>> After reading Julia docs on types it is still unclear for me how how to 
>>> use singleton classes for common tasks, e.g. for packaging some common 
>>> values.
>>>
>>> As a side note let me elaborate with the example from Scala programming 
>>> language.
>>> Scala has intrinsic support for singletons - Objects.
>>>
>>> ```scala
>>> scala> object CParam {
>>>  | var rho: Double = 1.0
>>>  | var cc: Double = 1.0
>>>  | }
>>> defined module CParam
>>>
>>> scala> CParam.rho = 2.0
>>> CParam.rho: Double = 2.0
>>>
>>> scala> import CParam._
>>> import CParam._
>>>
>>> scala> rho
>>> res0: Double = 2.0
>>> ```
>>>
>>> Now trying to project it to Julia:
>>>
>>> ```jlcon
>>> julia> type CParam
>>>  rho::Float64
>>>  cc::Float64
>>>  function CParam()
>>>new(0.0, 0.0)
>>>  end
>>>end
>>>
>>> julia> x = CParam()
>>> CParam(0.0,0.0)
>>>
>>> julia> x.rho
>>> 0.0
>>> ```
>>>
>>> So good so far, though this is not a singleton, just an instance.
>>> The Julia documentation says that parametric Type{T} is a special 
>>> parametric kind of the type T - the singleton type.
>>>
>>>
>>> ```jlcon
>>> julia> x = Type{CParam}
>>> Type{CParam}
>>>
>>> julia> x.rho
>>> ERROR: type DataType has no field rho
>>> ```
>>>
>>> Trying to exploit modules as model for singleton class (follow Fortran90 
>>> - it uses modules for information hiding and packaging).
>>>
>>> ```jlcon
>>> julia> module CParam
>>> export rho, bulk, cc, zz
>>> rho  = 1.0 
>>> bulk = 1.0 
>>> cc   = 1.0 
>>> zz   = 1.0 
>>>end
>>>  
>>> julia> CParam.rho  = 2.0
>>> ERROR: cannot assign variables in other modules
>>> ```
>>> Could you please recommend a programming idiom better fitting to Julia?
>>>
>>> Thanks!
>>> Alexander
>>>
>>>
>>

[julia-users] Re: ERROR: no method getindex(Array{Any,1},ASCIIString)

[Ivar Nesje]

The filename  /Applications/JuliaStudio.app/Contents/Resources/juliet/src/
modules/network/network.jl suggests OSX.

kl. 17:31:49 UTC+1 tirsdag 21. januar 2014 skrev Kees van Prooijen følgende:
>
> Sorry, I cannot reproduce this behavior.
> I could imagine you having problems the other way round, entering 
> multi-line input in the console. We made fixes for several problems there 
> that will be released soon.
> What platform are you on?
>
> Kees
>
> On Tuesday, January 21, 2014 6:56:34 AM UTC-8, Ben Simmons wrote:
>>
>> I've just installed Julia Studio last night to try and do some work on 
>> cellular automata.
>>
>> I've been familiarising myself with the language, and so far it's worked 
>> fine. But when I try to use the if command, I get the following error:
>>
>> ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
>>  
>> in Message at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
>>  
>> in process_incoming at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
>>  
>> in anonymous at multi.jl:110
>>
>> This is when running even very simple code, such as:
>>
>> x=2
>> y=4
>> if x < y
>>   println("x is less than y")
>> elseif x > y
>>   println("x is greater than y")
>> else
>>   println("x is equal to y")
>> end
>>
>> Or
>>
>> x=2
>> if x == 2
>>   println("x is 2")
>> end
>>
>> Any ideas? I also can't get it to work in Julia (rather than Julia Studio)
>>
>

Re: [julia-users] Type Conversion and Promotion for 'in' operator

[Sharmila Gopirajan]

Thanks for the heads up.  I will use the master then. I am still interested
in implementing the hashing strategy for numbers.  So any feedback would be
great.

Regards,
Sharmi


On Tue, Jan 21, 2014 at 10:53 PM, Milan Bouchet-Valat wrote:

>  Le mardi 21 janvier 2014 à 00:13 -0500, Jeff Bezanson a écrit :
>
> The main reason is that there are many types of numbers, with more
> added all the time. And for purposes of hash tables, it is difficult
> to ensure that all numerically-equal numbers hash the same. So we had
> isequal(), which is used by dictionaries, distinguish numbers of
> different types. At this point, we would kind of like to change this
> back and make isequal more liberal (although it would still
> distinguish -0.0 and 0.0, and so not be strictly more liberal than
> ==). However, the hashing problem remains. Any ideas are welcome.
>
>  Actually, you changed the behavior of in to use == instead of isequal()after 
> I filed an issue:
> https://github.com/JuliaLang/julia/issues/4941
>
>
> With git master as of a few days, this works:
>
> julia> x = int32(4)
> 4
>
> julia> y = int64(4)
> 4
>
> julia> x == y
> true
>
> julia> x in [y]
> true
>
> That doesn't mean hashing shouldn't be improved, though.
>
>
> Regards
>

Re: [julia-users] Type Conversion and Promotion for 'in' operator

[Milan Bouchet-Valat]

Le mardi 21 janvier 2014 à 00:13 -0500, Jeff Bezanson a écrit :

> The main reason is that there are many types of numbers, with more
> added all the time. And for purposes of hash tables, it is difficult
> to ensure that all numerically-equal numbers hash the same. So we had
> isequal(), which is used by dictionaries, distinguish numbers of
> different types. At this point, we would kind of like to change this
> back and make isequal more liberal (although it would still
> distinguish -0.0 and 0.0, and so not be strictly more liberal than
> ==). However, the hashing problem remains. Any ideas are welcome.

Actually, you changed the behavior of in to use == instead of isequal()
after I filed an issue: https://github.com/JuliaLang/julia/issues/4941


With git master as of a few days, this works:
julia> x = int32(4)
4

julia> y = int64(4)
4

julia> x == y
true

julia> x in [y]
true

That doesn't mean hashing shouldn't be improved, though.


Regards

Re: [julia-users] New Year's resolutions for DataArrays, DataFrames and other packages

[John Myles White]

Can you do something like df[“ColA”] = f(df)?

 — John

On Jan 21, 2014, at 8:48 AM, Blake Johnson  wrote:

> I use within! pretty frequently. What should I be using instead if that is on 
> the chopping block?
> 
> --Blake
> 
> On Tuesday, January 21, 2014 7:42:39 AM UTC-5, tshort wrote:
> I also agree with your approach, John. Based on your criteria, here 
> are some other things to consider for the chopping block. 
> 
> - expression-based indexing 
> - NamedArray (you already have an issue on this) 
> - with, within, based_on and variants 
> - @transform, @DataFrame 
> - select, filter 
> - DataStream 
> 
> Many of these were attempts to ease syntax via delayed evaluation. We 
> can either do without or try to implement something like LINQ. 
> 
> 
> 
> On Mon, Jan 20, 2014 at 7:02 PM, Kevin Squire  wrote: 
> > Hi John, 
> > 
> > I agree with pretty much everything you have written here, and really 
> > appreciate that you've taken the lead in cleaning things up and getting us 
> > on track. 
> > 
> > Cheers! 
> >Kevin 
> > 
> > 
> > On Mon, Jan 20, 2014 at 1:57 PM, John Myles White  
> > wrote: 
> >> 
> >> As I said in another thread recently, I am currently the lead maintainer 
> >> of more packages than I can keep up with. I think it’s been useful for me 
> >> to 
> >> start so many different projects, but I can’t keep maintaining most of my 
> >> packages given my current work schedule. 
> >> 
> >> Without Simon Kornblith, Kevin Squire, Sean Garborg and several others 
> >> doing amazing work to keep DataArrays and DataFrames going, much of our 
> >> basic data infrastructure would have already become completely unusable. 
> >> But 
> >> even with the great work that’s been done on those package recently, 
> >> there’s 
> >> still lot of additional design work required. I’d like to free up some of 
> >> my 
> >> time to do that work. 
> >> 
> >> To keep things moving forward, I’d like to propose a couple of radical New 
> >> Year’s resolutions for the packages I work on. 
> >> 
> >> (1) We need to stop adding functionality and focus entirely on improving 
> >> the quality and documentation of our existing functionality. We have way 
> >> too 
> >> much prototype code in DataFrames that I can’t keep up with. I’m about to 
> >> make a pull request for DataFrames that will remove everything related to 
> >> column groupings, database-style indexing and Blocks.jl support. I 
> >> absolutely want to see us push all of those ideas forward in the future, 
> >> but 
> >> they need to happen in unmerged forks or separate packages until we have 
> >> the 
> >> resources needed to support them. Right now, they make an overwhelming 
> >> maintenance challenge even more onerous. 
> >> 
> >> (2) We can’t support anything other than the master branch of most 
> >> JuliaStats packages except possibly for Distributions. I personally don’t 
> >> have the time to simultaneously keep stuff working with Julia 0.2 and 
> >> Julia 
> >> 0.3. Moreover, many of our basic packages aren’t mature enough to justify 
> >> supporting older versions. We should do a better job of supporting our 
> >> master releases and not invest precious time trying to support older 
> >> releases. 
> >> 
> >> (3) We need to make more of DataArrays and DataFrames reflect the Julian 
> >> worldview. Lots of our code uses an interface that is incongruous with the 
> >> interfaces found in Base. Even worse, a large chunk of code has 
> >> type-stability problems that makes it very slow, when comparable code that 
> >> uses normal Arrays is 100x faster. We need to develop new idioms and new 
> >> strategies for making code that interacts with type-destabilizing NA’s 
> >> faster. More generally, we need to make DataArrays and DataFrames fit in 
> >> better with Julia when Julia and R disagree. Following R’s lead has often 
> >> lead us astray because R doesn’t share Julia’s strenths or weaknesses. 
> >> 
> >> (4) Going forward, there should be exactly one way to do most things. The 
> >> worst part of our current codebase is that there are multiple ways to 
> >> express the same computation, but (a) some of them are unusably slow and 
> >> (b) 
> >> some of them don’t ever get tested or maintained properly. This is closely 
> >> linked to the excess proliferation of functionality described in 
> >> Resolution 
> >> 1 above. We need to start removing stuff from our packages and making the 
> >> parts we keep both reliable and fast. 
> >> 
> >> I think we can push DataArrays and DataFrames to 1.0 status by the end of 
> >> this year. But I think we need to adopt a new approach if we’re going to 
> >> get 
> >> there. Lots of stuff needs to get deprecated and what remains needs a lot 
> >> more testing, benchmarking and documentation. 
> >> 
> >>  — John 
> >> 
> >

Re: [julia-users] Type Conversion and Promotion for 'in' operator

[Sharmila Gopirajan]

Hi Jeff,
   Here is a strategy inspired by the hashing function in Python and
Julia.  I hope it covers all bases in integers, float, rationals, complex
and arbitrary precision. Please go through it and provide your feedback.

Please note that the code is supposed to be pseudocode and it is not
tested. also hash64 is a function already implemented in Julia.

hash
(x::Union(Bool,Char,Int8,Uint8,Int16,Uint16,Int32,Uint32,Int64,Uint64))=
hash64(uint64(x))

hash(x::Union(Float16, Float32, Rational{T<:Integer})) = hash(float64(x))

function hash(x::Union(Int128, Uint128))
x = uint128(x)
 x_one = uint64(x>>64)
x_two = uint64((x<<64)>>64))
 hash((x_one, x_two))

function hash(x::Float64)
 integral, fracpart = modf(x) #C function call
if fracpart == 0
 hash(integral)
else
 #custom hashing algo for float taking fracpart into account
end

function hash(x::Complex{T<:Real})
if imag(x) ==0
hash(real(x))
else
hash(real(x), real(y))
end
end

function hash(x::BigInt)
if typemin(Int128) <= x <= typemax(Int128)
return hash(int128(x))
else:
return hash(string(x))
end
end

function hash(x::BigFloat)
integral, fracpart = modf_equivalent_of_bigfloat(x)
if fracpart == 0
return hash(integral) #integral could be BigInt
else:
return hash(string(x))
end
end




On Tue, Jan 21, 2014 at 10:43 AM, Jeff Bezanson wrote:

> The main reason is that there are many types of numbers, with more
> added all the time. And for purposes of hash tables, it is difficult
> to ensure that all numerically-equal numbers hash the same. So we had
> isequal(), which is used by dictionaries, distinguish numbers of
> different types. At this point, we would kind of like to change this
> back and make isequal more liberal (although it would still
> distinguish -0.0 and 0.0, and so not be strictly more liberal than
> ==). However, the hashing problem remains. Any ideas are welcome.
>
>
> On Mon, Jan 20, 2014 at 11:54 PM, Sharmila Gopirajan Sivakumar
>  wrote:
> > julia> x = int32(4)
> > 4
> >
> > julia> y = int64(4)
> > 4
> >
> > julia> x == y
> > true
> >
> > julia> x in [y]
> > false
> >
> > I'm not sure if this behaviour is expected. Logically,  if x==y
> evaluates to
> > true, x in [y] should also evaluate to true.  The difference arises
> because
> > the implementation for in uses 'isequal' function to check for equality
> > instead of ==. 'isequal' is documented as follows.
> >
> > isequal(x, y)
> >
> > True if and only if x and y have the same contents. Loosely speaking,
> this
> > means x and y would look the same when printed. This is the default
> > comparison function used by hash tables (Dict). New types with a notion
> of
> > equality should implement this function, except for numbers, which should
> > implement == instead. However, numeric types with special values might
> need
> > to implement isequal as well. For example, floating point NaN values are
> not
> > ==, but are all equivalent in the sense of isequal. Numbers of different
> > types are considered unequal. Mutable containers should generally
> implement
> > isequal by calling isequal recursively on all contents.
> >
> > Why is isequal more liberal than == in some cases (like floating point
> NaNs)
> > and more strict than == is others (Numbers of different types are
> considered
> > unequal.).  In the above case, I believe it should evaluate to True, but
> I
> > assume there would be a good reason for it being the way it is.  Will
> > someone please clarify why?
>

Re: [julia-users] New Year's resolutions for DataArrays, DataFrames and other packages

[Blake Johnson]

I use within! pretty frequently. What should I be using instead if that is 
on the chopping block?

--Blake

On Tuesday, January 21, 2014 7:42:39 AM UTC-5, tshort wrote:
>
> I also agree with your approach, John. Based on your criteria, here 
> are some other things to consider for the chopping block. 
>
> - expression-based indexing 
> - NamedArray (you already have an issue on this) 
> - with, within, based_on and variants 
> - @transform, @DataFrame 
> - select, filter 
> - DataStream 
>
> Many of these were attempts to ease syntax via delayed evaluation. We 
> can either do without or try to implement something like LINQ. 
>
>
>
> On Mon, Jan 20, 2014 at 7:02 PM, Kevin Squire 
> > 
> wrote: 
> > Hi John, 
> > 
> > I agree with pretty much everything you have written here, and really 
> > appreciate that you've taken the lead in cleaning things up and getting 
> us 
> > on track. 
> > 
> > Cheers! 
> >Kevin 
> > 
> > 
> > On Mon, Jan 20, 2014 at 1:57 PM, John Myles White 
> > > 
>
> > wrote: 
> >> 
> >> As I said in another thread recently, I am currently the lead 
> maintainer 
> >> of more packages than I can keep up with. I think it’s been useful for 
> me to 
> >> start so many different projects, but I can’t keep maintaining most of 
> my 
> >> packages given my current work schedule. 
> >> 
> >> Without Simon Kornblith, Kevin Squire, Sean Garborg and several others 
> >> doing amazing work to keep DataArrays and DataFrames going, much of our 
> >> basic data infrastructure would have already become completely 
> unusable. But 
> >> even with the great work that’s been done on those package recently, 
> there’s 
> >> still lot of additional design work required. I’d like to free up some 
> of my 
> >> time to do that work. 
> >> 
> >> To keep things moving forward, I’d like to propose a couple of radical 
> New 
> >> Year’s resolutions for the packages I work on. 
> >> 
> >> (1) We need to stop adding functionality and focus entirely on 
> improving 
> >> the quality and documentation of our existing functionality. We have 
> way too 
> >> much prototype code in DataFrames that I can’t keep up with. I’m about 
> to 
> >> make a pull request for DataFrames that will remove everything related 
> to 
> >> column groupings, database-style indexing and Blocks.jl support. I 
> >> absolutely want to see us push all of those ideas forward in the 
> future, but 
> >> they need to happen in unmerged forks or separate packages until we 
> have the 
> >> resources needed to support them. Right now, they make an overwhelming 
> >> maintenance challenge even more onerous. 
> >> 
> >> (2) We can’t support anything other than the master branch of most 
> >> JuliaStats packages except possibly for Distributions. I personally 
> don’t 
> >> have the time to simultaneously keep stuff working with Julia 0.2 and 
> Julia 
> >> 0.3. Moreover, many of our basic packages aren’t mature enough to 
> justify 
> >> supporting older versions. We should do a better job of supporting our 
> >> master releases and not invest precious time trying to support older 
> >> releases. 
> >> 
> >> (3) We need to make more of DataArrays and DataFrames reflect the 
> Julian 
> >> worldview. Lots of our code uses an interface that is incongruous with 
> the 
> >> interfaces found in Base. Even worse, a large chunk of code has 
> >> type-stability problems that makes it very slow, when comparable code 
> that 
> >> uses normal Arrays is 100x faster. We need to develop new idioms and 
> new 
> >> strategies for making code that interacts with type-destabilizing NA’s 
> >> faster. More generally, we need to make DataArrays and DataFrames fit 
> in 
> >> better with Julia when Julia and R disagree. Following R’s lead has 
> often 
> >> lead us astray because R doesn’t share Julia’s strenths or weaknesses. 
> >> 
> >> (4) Going forward, there should be exactly one way to do most things. 
> The 
> >> worst part of our current codebase is that there are multiple ways to 
> >> express the same computation, but (a) some of them are unusably slow 
> and (b) 
> >> some of them don’t ever get tested or maintained properly. This is 
> closely 
> >> linked to the excess proliferation of functionality described in 
> Resolution 
> >> 1 above. We need to start removing stuff from our packages and making 
> the 
> >> parts we keep both reliable and fast. 
> >> 
> >> I think we can push DataArrays and DataFrames to 1.0 status by the end 
> of 
> >> this year. But I think we need to adopt a new approach if we’re going 
> to get 
> >> there. Lots of stuff needs to get deprecated and what remains needs a 
> lot 
> >> more testing, benchmarking and documentation. 
> >> 
> >>  — John 
> >> 
> > 
>

Re: [julia-users] Higher order derivatives in Calculus

[Hans W Borchers]

Thanks for these encouraging words. I have already written an R package 
with more than a hundred numerical functions (incl. several numerical 
derivatives), and I would be willing to help build up a numerical package 
in Julia. But of course, someone from the Julia community will be needed to 
take the lead. Please let me know when this 'management position'(?) has 
been taken.

On Tuesday, January 21, 2014 4:44:37 PM UTC+1, John Myles White wrote:
>
> Just to chime in: the biggest problem with the Calculus isn’t the absence 
> of usable functionality, it’s that the published interface isn’t a very 
> good one and the more reliable interface, including things like 
> finite_difference_hessian, isn’t exported. 
>
> To fix this, we need someone to come in and do some serious design work, 
> where they'll rethink interfaces and remove out-dated functionality. As Tim 
> Holy mentioned, the combination of the unpublished finite diference methods 
> and automatic differentation methods in DualNumbers should get you very 
> far. 
>
>  — John 
>
>

Re: [julia-users] Q: tutorial for working with Julla singleton types

[Alexander Samoilov]

Ok, thanks.

I got the idea.

On Tuesday, January 21, 2014 8:31:19 PM UTC+4, Stefan Karpinski wrote:
>
> Julia's singleton types are abstract types whose only instance is a single 
> type object. This is useful for writing methods that dispatch on type 
> values rather than on the type of an argument. They are abstract types – 
> they don't have fields and they don't hold data.
>
> If you want an object to store some values, define a type and make a 
> single instance of it:
>
> type CParam
>   rho::Float64
>   cc::Float64
> end
> const cparam = CParam(0,0)
>
>
> Now use cparam as your "singleton" – if you don't want more than one 
> instance, don't create more than one.
>
>
>
> On Tue, Jan 21, 2014 at 10:57 AM, Alexander Samoilov <
> alexander...@googlemail.com > wrote:
>
>> Hello Experts!
>>
>> After reading Julia docs on types it is still unclear for me how how to 
>> use singleton classes for common tasks, e.g. for packaging some common 
>> values.
>>
>> As a side note let me elaborate with the example from Scala programming 
>> language.
>> Scala has intrinsic support for singletons - Objects.
>>
>> ```scala
>> scala> object CParam {
>>  | var rho: Double = 1.0
>>  | var cc: Double = 1.0
>>  | }
>> defined module CParam
>>
>> scala> CParam.rho = 2.0
>> CParam.rho: Double = 2.0
>>
>> scala> import CParam._
>> import CParam._
>>
>> scala> rho
>> res0: Double = 2.0
>> ```
>>
>> Now trying to project it to Julia:
>>
>> ```jlcon
>> julia> type CParam
>>  rho::Float64
>>  cc::Float64
>>  function CParam()
>>new(0.0, 0.0)
>>  end
>>end
>>
>> julia> x = CParam()
>> CParam(0.0,0.0)
>>
>> julia> x.rho
>> 0.0
>> ```
>>
>> So good so far, though this is not a singleton, just an instance.
>> The Julia documentation says that parametric Type{T} is a special 
>> parametric kind of the type T - the singleton type.
>>
>>
>> ```jlcon
>> julia> x = Type{CParam}
>> Type{CParam}
>>
>> julia> x.rho
>> ERROR: type DataType has no field rho
>> ```
>>
>> Trying to exploit modules as model for singleton class (follow Fortran90 
>> - it uses modules for information hiding and packaging).
>>
>> ```jlcon
>> julia> module CParam
>> export rho, bulk, cc, zz
>> rho  = 1.0 
>> bulk = 1.0 
>> cc   = 1.0 
>> zz   = 1.0 
>>end
>>  
>> julia> CParam.rho  = 2.0
>> ERROR: cannot assign variables in other modules
>> ```
>> Could you please recommend a programming idiom better fitting to Julia?
>>
>> Thanks!
>> Alexander
>>
>>
>

[julia-users] Re: ERROR: no method getindex(Array{Any,1},ASCIIString)

[Kees van Prooijen]

Sorry, I cannot reproduce this behavior.
I could imagine you having problems the other way round, entering 
multi-line input in the console. We made fixes for several problems there 
that will be released soon.
What platform are you on?

Kees

On Tuesday, January 21, 2014 6:56:34 AM UTC-8, Ben Simmons wrote:
>
> I've just installed Julia Studio last night to try and do some work on 
> cellular automata.
>
> I've been familiarising myself with the language, and so far it's worked 
> fine. But when I try to use the if command, I get the following error:
>
> ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
>  
> in Message at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
>  
> in process_incoming at 
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
>  
> in anonymous at multi.jl:110
>
> This is when running even very simple code, such as:
>
> x=2
> y=4
> if x < y
>   println("x is less than y")
> elseif x > y
>   println("x is greater than y")
> else
>   println("x is equal to y")
> end
>
> Or
>
> x=2
> if x == 2
>   println("x is 2")
> end
>
> Any ideas? I also can't get it to work in Julia (rather than Julia Studio)
>

Re: [julia-users] Q: tutorial for working with Julla singleton types

[Stefan Karpinski]

Julia's singleton types are abstract types whose only instance is a single
type object. This is useful for writing methods that dispatch on type
values rather than on the type of an argument. They are abstract types –
they don't have fields and they don't hold data.

If you want an object to store some values, define a type and make a single
instance of it:

type CParam
  rho::Float64
  cc::Float64
end
const cparam = CParam(0,0)


Now use cparam as your "singleton" – if you don't want more than one
instance, don't create more than one.



On Tue, Jan 21, 2014 at 10:57 AM, Alexander Samoilov <
alexander.samoi...@googlemail.com> wrote:

> Hello Experts!
>
> After reading Julia docs on types it is still unclear for me how how to
> use singleton classes for common tasks, e.g. for packaging some common
> values.
>
> As a side note let me elaborate with the example from Scala programming
> language.
> Scala has intrinsic support for singletons - Objects.
>
> ```scala
> scala> object CParam {
>  | var rho: Double = 1.0
>  | var cc: Double = 1.0
>  | }
> defined module CParam
>
> scala> CParam.rho = 2.0
> CParam.rho: Double = 2.0
>
> scala> import CParam._
> import CParam._
>
> scala> rho
> res0: Double = 2.0
> ```
>
> Now trying to project it to Julia:
>
> ```jlcon
> julia> type CParam
>  rho::Float64
>  cc::Float64
>  function CParam()
>new(0.0, 0.0)
>  end
>end
>
> julia> x = CParam()
> CParam(0.0,0.0)
>
> julia> x.rho
> 0.0
> ```
>
> So good so far, though this is not a singleton, just an instance.
> The Julia documentation says that parametric Type{T} is a special
> parametric kind of the type T - the singleton type.
>
>
> ```jlcon
> julia> x = Type{CParam}
> Type{CParam}
>
> julia> x.rho
> ERROR: type DataType has no field rho
> ```
>
> Trying to exploit modules as model for singleton class (follow Fortran90 -
> it uses modules for information hiding and packaging).
>
> ```jlcon
> julia> module CParam
> export rho, bulk, cc, zz
> rho  = 1.0
> bulk = 1.0
> cc   = 1.0
> zz   = 1.0
>end
>
> julia> CParam.rho  = 2.0
> ERROR: cannot assign variables in other modules
> ```
> Could you please recommend a programming idiom better fitting to Julia?
>
> Thanks!
> Alexander
>
>

Re: [julia-users] New Year's resolutions for DataArrays, DataFrames and other packages

[John Myles White]

I agree with everything on this list, including my always neglected DataStreams 
project.

I think it would be nice to get rid of expression-based indexing + select and 
focus on getting something like LINQ working. For another interesting 
perspective, check out the nearly created query function in Pandas, which takes 
in strings rather than expressions as inputs.

 — John

On Jan 21, 2014, at 4:42 AM, Tom Short  wrote:

> I also agree with your approach, John. Based on your criteria, here
> are some other things to consider for the chopping block.
> 
> - expression-based indexing
> - NamedArray (you already have an issue on this)
> - with, within, based_on and variants
> - @transform, @DataFrame
> - select, filter
> - DataStream
> 
> Many of these were attempts to ease syntax via delayed evaluation. We
> can either do without or try to implement something like LINQ.

[julia-users] Q: tutorial for working with Julla singleton types

[Alexander Samoilov]

Hello Experts!

After reading Julia docs on types it is still unclear for me how how to use 
singleton classes for common tasks, e.g. for packaging some common values.

As a side note let me elaborate with the example from Scala programming 
language.
Scala has intrinsic support for singletons - Objects.

```scala
scala> object CParam {
 | var rho: Double = 1.0
 | var cc: Double = 1.0
 | }
defined module CParam

scala> CParam.rho = 2.0
CParam.rho: Double = 2.0

scala> import CParam._
import CParam._

scala> rho
res0: Double = 2.0
```

Now trying to project it to Julia:

```jlcon
julia> type CParam
 rho::Float64
 cc::Float64
 function CParam()
   new(0.0, 0.0)
 end
   end

julia> x = CParam()
CParam(0.0,0.0)

julia> x.rho
0.0
```

So good so far, though this is not a singleton, just an instance.
The Julia documentation says that parametric Type{T} is a special 
parametric kind of the type T - the singleton type.


```jlcon
julia> x = Type{CParam}
Type{CParam}

julia> x.rho
ERROR: type DataType has no field rho
```

Trying to exploit modules as model for singleton class (follow Fortran90 - 
it uses modules for information hiding and packaging).

```jlcon
julia> module CParam
export rho, bulk, cc, zz
rho  = 1.0 
bulk = 1.0 
cc   = 1.0 
zz   = 1.0 
   end
 
julia> CParam.rho  = 2.0
ERROR: cannot assign variables in other modules
```
Could you please recommend a programming idiom better fitting to Julia?

Thanks!
Alexander

Re: [julia-users] Higher order derivatives in Calculus

[John Myles White]

Just to chime in: the biggest problem with the Calculus isn’t the absence of 
usable functionality, it’s that the published interface isn’t a very good one 
and the more reliable interface, including things like 
finite_difference_hessian, isn’t exported.

To fix this, we need someone to come in and do some serious design work, where 
they'll rethink interfaces and remove out-dated functionality. As Tim Holy 
mentioned, the combination of the unpublished finite diference methods and 
automatic differentation methods in DualNumbers should get you very far.

 — John

On Jan 21, 2014, at 7:20 AM, Tim Holy  wrote:

> On Tuesday, January 21, 2014 05:32:13 AM Hans W Borchers wrote:
>> When you say, Calculus is not developed much at the moment,
>> maybe it's too early for me to change.
> 
> Writing finite-differencing algorithms isn't that hard. That should not be a 
> make-or-break issue for your decision about whether to use Julia.
> 
> But don't underestimate the automatic differentiation facilities that have 
> recently been added to Julia (https://github.com/scidom/DualNumbers.jl). 
> Basically, AD computes numerical derivatives without the roundoff error, by 
> defining a new numerical type that behaves somewhat similarly to complex 
> numbers but extracts the first derivative exactly. The key point is that it 
> is 
> a _numerical_ approach, so it doesn't rely on anything symbolic. The one 
> place 
> you can't use AD is when your function relies on calling out to C (because C 
> doesn't know about Julia's Dual type). But any function defined in Julia, 
> including special functions like elliptic integrals, etc, should be fine.
> 
> For higher-order derivatives, you can do similar things with even more fancy 
> numerical types. Perhaps the new PowerSeries already does this? (I haven't 
> looked.)
> 
> --Tim
> 

Re: [julia-users] Re: Higher order derivatives in Calculus

[Tim Holy]

On Tuesday, January 21, 2014 05:32:13 AM Hans W Borchers wrote:
> When you say, Calculus is not developed much at the moment,
> maybe it's too early for me to change.

Writing finite-differencing algorithms isn't that hard. That should not be a 
make-or-break issue for your decision about whether to use Julia.

But don't underestimate the automatic differentiation facilities that have 
recently been added to Julia (https://github.com/scidom/DualNumbers.jl). 
Basically, AD computes numerical derivatives without the roundoff error, by 
defining a new numerical type that behaves somewhat similarly to complex 
numbers but extracts the first derivative exactly. The key point is that it is 
a _numerical_ approach, so it doesn't rely on anything symbolic. The one place 
you can't use AD is when your function relies on calling out to C (because C 
doesn't know about Julia's Dual type). But any function defined in Julia, 
including special functions like elliptic integrals, etc, should be fine.

For higher-order derivatives, you can do similar things with even more fancy 
numerical types. Perhaps the new PowerSeries already does this? (I haven't 
looked.)

--Tim

Re: [julia-users] ERROR: no method getindex(Array{Any,1},ASCIIString)

[Ben Simmons]

Huh.

It works if entered directly into the console, but not if run from the 
script editor.

On Tuesday, 21 January 2014 14:59:33 UTC, Stefan Karpinski wrote:
>
> I don't think the issue is with if – it's with I/O. Can you do 
> println("hello") by itself?
>
>
> On Tue, Jan 21, 2014 at 9:56 AM, Ben Simmons 
> > wrote:
>
>> I've just installed Julia Studio last night to try and do some work on 
>> cellular automata.
>>
>> I've been familiarising myself with the language, and so far it's worked 
>> fine. But when I try to use the if command, I get the following error:
>>
>> ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
>>  
>> in Message at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
>>  
>> in process_incoming at 
>> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
>>  
>> in anonymous at multi.jl:110
>>
>> This is when running even very simple code, such as:
>>
>> x=2
>> y=4
>> if x < y
>>   println("x is less than y")
>> elseif x > y
>>   println("x is greater than y")
>> else
>>   println("x is equal to y")
>> end
>>
>> Or
>>
>> x=2
>> if x == 2
>>   println("x is 2")
>> end
>>
>> Any ideas? I also can't get it to work in Julia (rather than Julia Studio)
>>
>
>

Re: [julia-users] Pkg.add("...") fails for Gadfly&Co on Windows 8.

[Patrick O'Leary]

git config --global --unset url."https://github.com/".insteadOf

(or)

git config --global --edit

On Tuesday, January 21, 2014 8:44:18 AM UTC-6, Stefan Karpinski wrote:
>
> Looks like you're having a DNS issue. I'm not sure how to fix it, however. 
> One thing you could try is doing
>
> git config --global url."https://github.com/".insteadOf "git://github.com/
> "
>
> at a terminal and then see if that fixes things. That will modify your 
> global git config file, so to go back to a normal state, you will need to 
> edit that file, but I have no idea where it might live on Windows.
>
>
> On Tue, Jan 21, 2014 at 4:18 AM, Raffael Vogler 
> 
> > wrote:
>
>> Hi!
>>
>> Trying to install Winston, Gadfly or Gaston as suggested here 
>> fails with following error message:
>>
>> julia> Pkg.add("Gadfly")
>> Pkg.add("Gadfly")
>> INFO: Initializing package repository C:\Users\Raffael\.julia
>> INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
>> fatal: Unable to look up github.com (port 9418) (Beim Datenbankaufrufist ein 
>> ni
>> cht behebbarer Fehler aufgetreten.)
>> ERROR: failed process: Process(`git clone -q -b metadata-v2 git://
>> github.com/Jul
>> iaLang/METADATA.jl METADATA`, ProcessExited(128)) [128]
>>  in pipeline_error at process.jl:476
>>  in success at process.jl:468
>>  in run at process.jl:453
>>
>>
>> (Beim Datenbankaufruf ist ein nicht behebbarer Fehler aufgetreten.) 
>>  = (During the data base request a non-recoverable error occured.)
>>
>> I just joined the community one hour ago and am pretty clueless how to 
>> deal with this issue.
>>
>> Julia:
>> Version 0.2.0 (2013-11-16 23:44 UTC)
>> Official http://julialang.org release
>> x86_64-w64-mingw32
>>
>> OS:
>> Windows 8 Pro 64bit
>>
>> Kind regards
>>
>> Raffael
>>
>
>

Re: [julia-users] ERROR: no method getindex(Array{Any,1},ASCIIString)

[Stefan Karpinski]

I don't think the issue is with if – it's with I/O. Can you do
println("hello") by itself?


On Tue, Jan 21, 2014 at 9:56 AM, Ben Simmons wrote:

> I've just installed Julia Studio last night to try and do some work on
> cellular automata.
>
> I've been familiarising myself with the language, and so far it's worked
> fine. But when I try to use the if command, I get the following error:
>
> ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
> in Message at
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
> in process_incoming at
> /Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
> in anonymous at multi.jl:110
>
> This is when running even very simple code, such as:
>
> x=2
> y=4
> if x < y
>   println("x is less than y")
> elseif x > y
>   println("x is greater than y")
> else
>   println("x is equal to y")
> end
>
> Or
>
> x=2
> if x == 2
>   println("x is 2")
> end
>
> Any ideas? I also can't get it to work in Julia (rather than Julia Studio)
>

[julia-users] ERROR: no method getindex(Array{Any,1},ASCIIString)

[Ben Simmons]

I've just installed Julia Studio last night to try and do some work on 
cellular automata.

I've been familiarising myself with the language, and so far it's worked 
fine. But when I try to use the if command, I get the following error:

ERROR: no method getindex(Array{Any,1},ASCIIString) in read_message at 
/Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:45
 
in Message at 
/Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:19
 
in process_incoming at 
/Applications/JuliaStudio.app/Contents/Resources/juliet/src/modules/network/network.jl:59
 
in anonymous at multi.jl:110

This is when running even very simple code, such as:

x=2
y=4
if x < y
  println("x is less than y")
elseif x > y
  println("x is greater than y")
else
  println("x is equal to y")
end

Or

x=2
if x == 2
  println("x is 2")
end

Any ideas? I also can't get it to work in Julia (rather than Julia Studio)

Re: [julia-users] Pkg.add("...") fails for Gadfly&Co on Windows 8.

[Stefan Karpinski]

Looks like you're having a DNS issue. I'm not sure how to fix it, however.
One thing you could try is doing

git config --global url."https://github.com/".insteadOf "git://github.com/"

at a terminal and then see if that fixes things. That will modify your
global git config file, so to go back to a normal state, you will need to
edit that file, but I have no idea where it might live on Windows.


On Tue, Jan 21, 2014 at 4:18 AM, Raffael Vogler  wrote:

> Hi!
>
> Trying to install Winston, Gadfly or Gaston as suggested here
> fails with following error message:
>
> julia> Pkg.add("Gadfly")
> Pkg.add("Gadfly")
> INFO: Initializing package repository C:\Users\Raffael\.julia
> INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
> fatal: Unable to look up github.com (port 9418) (Beim Datenbankaufruf ist
> ein ni
> cht behebbarer Fehler aufgetreten.)
> ERROR: failed process: Process(`git clone -q -b metadata-v2 git://
> github.com/Jul
> iaLang/METADATA.jl METADATA`, ProcessExited(128)) [128]
>  in pipeline_error at process.jl:476
>  in success at process.jl:468
>  in run at process.jl:453
>
>
> (Beim Datenbankaufruf ist ein nicht behebbarer Fehler aufgetreten.)
>  = (During the data base request a non-recoverable error occured.)
>
> I just joined the community one hour ago and am pretty clueless how to
> deal with this issue.
>
> Julia:
> Version 0.2.0 (2013-11-16 23:44 UTC)
> Official http://julialang.org release
> x86_64-w64-mingw32
>
> OS:
> Windows 8 Pro 64bit
>
> Kind regards
>
> Raffael
>

[julia-users] Pkg.add("...") fails for Gadfly&Co on Windows 8.

[Raffael Vogler]

Hi!

Trying to install Winston, Gadfly or Gaston as suggested here 
fails with following error message:

julia> Pkg.add("Gadfly")
Pkg.add("Gadfly")
INFO: Initializing package repository C:\Users\Raffael\.julia
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
fatal: Unable to look up github.com (port 9418) (Beim Datenbankaufruf ist 
ein ni
cht behebbarer Fehler aufgetreten.)
ERROR: failed process: Process(`git clone -q -b metadata-v2 
git://github.com/Jul
iaLang/METADATA.jl METADATA`, ProcessExited(128)) [128]
 in pipeline_error at process.jl:476
 in success at process.jl:468
 in run at process.jl:453


(Beim Datenbankaufruf ist ein nicht behebbarer Fehler aufgetreten.) 
 = (During the data base request a non-recoverable error occured.)

I just joined the community one hour ago and am pretty clueless how to deal 
with this issue.

Julia:
Version 0.2.0 (2013-11-16 23:44 UTC)
Official http://julialang.org release
x86_64-w64-mingw32

OS:
Windows 8 Pro 64bit

Kind regards

Raffael

[julia-users] Re: Higher order derivatives in Calculus

[Hans W Borchers]

Ivar, if I sounded impolite, I feel sorry. And I don't want to 'criticize' 
the Calculus package, I am just trying to understand the status of the 
package and where it is going. I'd love to use Julia as a technical 
computing platform in the future, but for this I would require a strong 
calculus or 'NumericalMath' package that covers lots of numerical 
functions. When you say, Calculus is not developed much at the moment, 
maybe it's too early for me to change.

Hans Werner


On Tuesday, January 21, 2014 2:02:49 PM UTC+1, Ivar Nesje wrote:
>
> When I came here, I thought Julia was meant for "technical computing" and 
>> not so much for pure Math exercises. In scientific computing, calculating 
>> numerical derivatives and gradients is essential for many applications.
>
>
> Julia is not meant to be unusable for pure Math exercises. It is designed 
> to be free, and it is early in its development so things change as more 
> things gets implemented and new opinions are expressed. I have followed the 
> development for 7 months, and I am really amazed how polite the discussion 
> is, and how almost every question, hard or stupid, gets a (mostly) useful 
> response.
>
> The package you criticize is not part of the standard library, and 
> currently it is more of a "proof of concept" package, than anything else. 
> There seems to be lots of interest in using Calculus.jl, but unfortunately 
> the development of that package is missing a strong lead. In the last 7 
> months only small bugfixes has been added to that repository. 
>
> If you want to define a Julia function in terms of a Newton iteration, I 
> would think that the best option is to implement your own derivative 
> approximation where you can control the step. I often find that iterative 
> solvers gives a non smooth response in the solution, and a finite 
> difference over a small step will be very inaccurate.
>
> Ivar
>
> kl. 13:05:15 UTC+1 tirsdag 21. januar 2014 skrev Hans W Borchers følgende:
>>
>> Think of a special function not (yet) implemented in Julia, like Lambert 
>> W, or Struve, or ... So you define it yourself, e.g. by applying 
>> Newton-Raphson. How would you symbolically differentiate that? Or your 
>> application forces you to define a procedure with if-else constructs and 
>> loops, how to differentiate? Even an advanced "automated differentiation" 
>> approach will most often not be capable of solving this for you.
>>
>> Or imagine having to minimize a function such as
>>
>> function repelling_balls(x::Array{Float64, 1})
>>   if length(x) != 45; error("Arg. 'x' must be a vector of length 45."); 
>> end
>>   d = 2.0
>>   for i = 1:14, j = (i+1):15
>>   s = sqrt((x[i]-x[j])^2 + (x[i+15]-x[j+15])^2 + (x[i+30]-x[j+30])^2)
>>   if s < d; d = s; end
>>   end
>>   return -d
>> end
>>
>> that models 15 repelling balls/points in [0, 1]^3.
>>
>> As "minimax" problem this function is smooth except for a subset of lower 
>> dimension. Some optimization procedures using gradients will be able to 
>> circumvent this subset and find a (local) minimum. How will you 
>> differentiate that (i.e., build a gradient) if not numerically; and if yes, 
>> is it worth the effort?
>>
>> Consider that an optimization or root finding procedure is a (hopefully) 
>> converging process that will not return an absolutely precise result. A 
>> function itself may be defined through an optimization or root finding task 
>> and whose derivative will have a technical meaning.
>>
>> In real-world applications, and tasks Matlab is made for, you do not care 
>> for 15 digits, and gradients five, six, seven digits accurate are more than 
>> enough to solve your problem. Normally, your functions are not defined for 
>> eps() accuracy anyway.
>>
>> When I came here, I thought Julia was meant for "technical computing" and 
>> not so much for pure Math exercises. In scientific computing, calculating 
>> numerical derivatives and gradients is essential for many applications.
>>
>>
>> On Tuesday, January 21, 2014 3:28:28 AM UTC+1, Jason Merrill wrote:
>>>
>>> Would you be willing to share a problem that you're interested in that 
>>> isn't amenable to symbolic differentiation, maybe on a gist or something? 
>>> I've been looking for non-trivial problems to try to apply PowerSeries to.
>>>
>>> When I said numerical differentiation is "unstable", I meant something 
>>> like "hard to do without losing several digits". For example, you point out 
>>> that by choosing a different h, it's possible to improve over Calculus.jl's 
>>> current behavior by 2-3 decimal digits, from 7e-7 to 2e-9 in absolute 
>>> error. But the best precision you could hope for in a floating point answer 
>>> to this problem is
>>>
>>> julia> eps(sin(1.0))
>>> 1.1102230246251565e-16
>>>
>>> so even with your improvements, you've lost a lot of precision. That 
>>> might or might not be a problem depending on what you'd like to do next 
>>> with the value you get out.
>>>
>>>
>>> On Monday, January

Re: [julia-users] Higher order derivatives in Calculus

[Ivar Nesje]

finite_difference_hessian is not an exported function in Calculus.jl (yet)

If you want to call it directly, you need to use the fully qualified name

Calculus.finite_difference_hessian



kl. 13:59:49 UTC+1 tirsdag 21. januar 2014 skrev Hans W Borchers følgende:
>
> Sorry, but I don't see it. Using Calculus it says
>
> julia> finite_difference_hessian(sin, 1.0)
> ERROR: finite_difference_hessian not defined
>
> and calling hessian alone returns the old, inaccurate result:
>
> julia> hessian(sin, 1.0)
> -0.841471649579559
>
> Looking at the definition of hessian, I'd say it calls
> "finite_difference_hessian(f, derivative(f), x, :central)"
> so basing the calculation on a derivative function and not on a formula 
> derived from the Taylor series.
>
> What did I misunderstand?
> Thanks, Hans Werner
>
>
> On Tuesday, January 21, 2014 3:52:37 AM UTC+1, Tim Holy wrote:
>>
>> Most of the second_derivative stuff in derivative.jl could probably be 
>> ripped 
>> out, and have it directly call finite_difference_hessian(f, x). Indeed, 
>> that 
>> gets you those two extra orders of magnitude in precision, because 
>> finite_difference_hessian uses exactly this kind of scaling rule. 
>>
>> --Tim 
>>
>> On Monday, January 20, 2014 10:40:28 AM Hans W Borchers wrote: 
>> > I looked into the *Calculus* package and its derivative functions. 
>> First, I 
>> > got errors when running examples from the README file: 
>> > 
>> > julia> second_derivative(x -> sin(x), pi) 
>> > ERROR: no method eps(DataType,) 
>> >  in finite_difference at 
>> > /Users/HwB/.julia/Calculus/src/finite_difference.jl:27 
>> >  in second_derivative at 
>> /Users/HwB/.julia/Calculus/src/derivative.jl:67 
>> > 
>> > Then I was a bit astonished to see not too accurate results such as 
>> > 
>> > julia> abs(second_derivative(sin, 1.0) + sin(1.0)) 
>> > 6.647716624952338e-7 
>> > 
>> > while, when applying the standard central formula for second 
>> derivatives, 
>> > (f(x+h) - 2*f(x) + f(x-h)) / h^2 with the (by theory) suggested step 
>> length 
>> > eps^0.25 (for second derivatives) will result in a much better value: 
>> > 
>> > julia> h = eps()^0.25; 
>> > 
>> > julia> f = sin; x = 1.0; 
>> > 
>> > julia> df = (sin(x+h) - 2*sin(x) + sin(x-h)) / h^2 
>> > -0.8414709866046906 
>> > 
>> > julia> abs(df + sin(1.0)) 
>> > 1.7967940468821553e-9 
>> > 
>> > The functions for numerical differentiation in *Calculus* look quite 
>> > involved, maybe it would be preferable to apply known approaches 
>> derived 
>> > from Taylor series. Even the fourth order derivative will in this case 
>> lead 
>> > to an absolute error below 1e-05! 
>>
>

[julia-users] Re: Higher order derivatives in Calculus

[Ivar Nesje]

>
> When I came here, I thought Julia was meant for "technical computing" and 
> not so much for pure Math exercises. In scientific computing, calculating 
> numerical derivatives and gradients is essential for many applications.


Julia is not meant to be unusable for pure Math exercises. It is designed 
to be free, and it is early in its development so things change as more 
things gets implemented and new opinions are expressed. I have followed the 
development for 7 months, and I am really amazed how polite the discussion 
is, and how almost every question, hard or stupid, gets a (mostly) useful 
response.

The package you criticize is not part of the standard library, and 
currently it is more of a "proof of concept" package, than anything else. 
There seems to be lots of interest in using Calculus.jl, but unfortunately 
the development of that package is missing a strong lead. In the last 7 
months only small bugfixes has been added to that repository. 

If you want to define a Julia function in terms of a Newton iteration, I 
would think that the best option is to implement your own derivative 
approximation where you can control the step. I often find that iterative 
solvers gives a non smooth response in the solution, and a finite 
difference over a small step will be very inaccurate.

Ivar

kl. 13:05:15 UTC+1 tirsdag 21. januar 2014 skrev Hans W Borchers følgende:
>
> Think of a special function not (yet) implemented in Julia, like Lambert 
> W, or Struve, or ... So you define it yourself, e.g. by applying 
> Newton-Raphson. How would you symbolically differentiate that? Or your 
> application forces you to define a procedure with if-else constructs and 
> loops, how to differentiate? Even an advanced "automated differentiation" 
> approach will most often not be capable of solving this for you.
>
> Or imagine having to minimize a function such as
>
> function repelling_balls(x::Array{Float64, 1})
>   if length(x) != 45; error("Arg. 'x' must be a vector of length 45."); end
>   d = 2.0
>   for i = 1:14, j = (i+1):15
>   s = sqrt((x[i]-x[j])^2 + (x[i+15]-x[j+15])^2 + (x[i+30]-x[j+30])^2)
>   if s < d; d = s; end
>   end
>   return -d
> end
>
> that models 15 repelling balls/points in [0, 1]^3.
>
> As "minimax" problem this function is smooth except for a subset of lower 
> dimension. Some optimization procedures using gradients will be able to 
> circumvent this subset and find a (local) minimum. How will you 
> differentiate that (i.e., build a gradient) if not numerically; and if yes, 
> is it worth the effort?
>
> Consider that an optimization or root finding procedure is a (hopefully) 
> converging process that will not return an absolutely precise result. A 
> function itself may be defined through an optimization or root finding task 
> and whose derivative will have a technical meaning.
>
> In real-world applications, and tasks Matlab is made for, you do not care 
> for 15 digits, and gradients five, six, seven digits accurate are more than 
> enough to solve your problem. Normally, your functions are not defined for 
> eps() accuracy anyway.
>
> When I came here, I thought Julia was meant for "technical computing" and 
> not so much for pure Math exercises. In scientific computing, calculating 
> numerical derivatives and gradients is essential for many applications.
>
>
> On Tuesday, January 21, 2014 3:28:28 AM UTC+1, Jason Merrill wrote:
>>
>> Would you be willing to share a problem that you're interested in that 
>> isn't amenable to symbolic differentiation, maybe on a gist or something? 
>> I've been looking for non-trivial problems to try to apply PowerSeries to.
>>
>> When I said numerical differentiation is "unstable", I meant something 
>> like "hard to do without losing several digits". For example, you point out 
>> that by choosing a different h, it's possible to improve over Calculus.jl's 
>> current behavior by 2-3 decimal digits, from 7e-7 to 2e-9 in absolute 
>> error. But the best precision you could hope for in a floating point answer 
>> to this problem is
>>
>> julia> eps(sin(1.0))
>> 1.1102230246251565e-16
>>
>> so even with your improvements, you've lost a lot of precision. That 
>> might or might not be a problem depending on what you'd like to do next 
>> with the value you get out.
>>
>>
>> On Monday, January 20, 2014 12:30:38 PM UTC-8, Hans W Borchers wrote:
>>>
>>> Numerical differentiation is by far not as unstable as you seem to think.
>>> And I have a long experience in using numerical derivatives for 
>>> optimization problems where you don't stop to look up symbolic derivatives 
>>> applying a CAS.
>>> The function obviously was only an example.
>>> For most of the functions I have used Julia's symbolic capabilities will 
>>> not be sufficient.
>>>
>>>
>>> On Monday, January 20, 2014 9:07:02 PM UTC+1, Jason Merrill wrote:

 This implementation could certainly use some love, but finite 
 difference differentiation is always unstable, and 

Re: [julia-users] Higher order derivatives in Calculus

[Hans W Borchers]

Sorry, but I don't see it. Using Calculus it says

julia> finite_difference_hessian(sin, 1.0)
ERROR: finite_difference_hessian not defined

and calling hessian alone returns the old, inaccurate result:

julia> hessian(sin, 1.0)
-0.841471649579559

Looking at the definition of hessian, I'd say it calls
"finite_difference_hessian(f, derivative(f), x, :central)"
so basing the calculation on a derivative function and not on a formula 
derived from the Taylor series.

What did I misunderstand?
Thanks, Hans Werner


On Tuesday, January 21, 2014 3:52:37 AM UTC+1, Tim Holy wrote:
>
> Most of the second_derivative stuff in derivative.jl could probably be 
> ripped 
> out, and have it directly call finite_difference_hessian(f, x). Indeed, 
> that 
> gets you those two extra orders of magnitude in precision, because 
> finite_difference_hessian uses exactly this kind of scaling rule. 
>
> --Tim 
>
> On Monday, January 20, 2014 10:40:28 AM Hans W Borchers wrote: 
> > I looked into the *Calculus* package and its derivative functions. 
> First, I 
> > got errors when running examples from the README file: 
> > 
> > julia> second_derivative(x -> sin(x), pi) 
> > ERROR: no method eps(DataType,) 
> >  in finite_difference at 
> > /Users/HwB/.julia/Calculus/src/finite_difference.jl:27 
> >  in second_derivative at 
> /Users/HwB/.julia/Calculus/src/derivative.jl:67 
> > 
> > Then I was a bit astonished to see not too accurate results such as 
> > 
> > julia> abs(second_derivative(sin, 1.0) + sin(1.0)) 
> > 6.647716624952338e-7 
> > 
> > while, when applying the standard central formula for second 
> derivatives, 
> > (f(x+h) - 2*f(x) + f(x-h)) / h^2 with the (by theory) suggested step 
> length 
> > eps^0.25 (for second derivatives) will result in a much better value: 
> > 
> > julia> h = eps()^0.25; 
> > 
> > julia> f = sin; x = 1.0; 
> > 
> > julia> df = (sin(x+h) - 2*sin(x) + sin(x-h)) / h^2 
> > -0.8414709866046906 
> > 
> > julia> abs(df + sin(1.0)) 
> > 1.7967940468821553e-9 
> > 
> > The functions for numerical differentiation in *Calculus* look quite 
> > involved, maybe it would be preferable to apply known approaches derived 
> > from Taylor series. Even the fourth order derivative will in this case 
> lead 
> > to an absolute error below 1e-05! 
>

Re: [julia-users] New Year's resolutions for DataArrays, DataFrames and other packages

[Tom Short]

I also agree with your approach, John. Based on your criteria, here
are some other things to consider for the chopping block.

- expression-based indexing
- NamedArray (you already have an issue on this)
- with, within, based_on and variants
- @transform, @DataFrame
- select, filter
- DataStream

Many of these were attempts to ease syntax via delayed evaluation. We
can either do without or try to implement something like LINQ.



On Mon, Jan 20, 2014 at 7:02 PM, Kevin Squire  wrote:
> Hi John,
>
> I agree with pretty much everything you have written here, and really
> appreciate that you've taken the lead in cleaning things up and getting us
> on track.
>
> Cheers!
>Kevin
>
>
> On Mon, Jan 20, 2014 at 1:57 PM, John Myles White 
> wrote:
>>
>> As I said in another thread recently, I am currently the lead maintainer
>> of more packages than I can keep up with. I think it’s been useful for me to
>> start so many different projects, but I can’t keep maintaining most of my
>> packages given my current work schedule.
>>
>> Without Simon Kornblith, Kevin Squire, Sean Garborg and several others
>> doing amazing work to keep DataArrays and DataFrames going, much of our
>> basic data infrastructure would have already become completely unusable. But
>> even with the great work that’s been done on those package recently, there’s
>> still lot of additional design work required. I’d like to free up some of my
>> time to do that work.
>>
>> To keep things moving forward, I’d like to propose a couple of radical New
>> Year’s resolutions for the packages I work on.
>>
>> (1) We need to stop adding functionality and focus entirely on improving
>> the quality and documentation of our existing functionality. We have way too
>> much prototype code in DataFrames that I can’t keep up with. I’m about to
>> make a pull request for DataFrames that will remove everything related to
>> column groupings, database-style indexing and Blocks.jl support. I
>> absolutely want to see us push all of those ideas forward in the future, but
>> they need to happen in unmerged forks or separate packages until we have the
>> resources needed to support them. Right now, they make an overwhelming
>> maintenance challenge even more onerous.
>>
>> (2) We can’t support anything other than the master branch of most
>> JuliaStats packages except possibly for Distributions. I personally don’t
>> have the time to simultaneously keep stuff working with Julia 0.2 and Julia
>> 0.3. Moreover, many of our basic packages aren’t mature enough to justify
>> supporting older versions. We should do a better job of supporting our
>> master releases and not invest precious time trying to support older
>> releases.
>>
>> (3) We need to make more of DataArrays and DataFrames reflect the Julian
>> worldview. Lots of our code uses an interface that is incongruous with the
>> interfaces found in Base. Even worse, a large chunk of code has
>> type-stability problems that makes it very slow, when comparable code that
>> uses normal Arrays is 100x faster. We need to develop new idioms and new
>> strategies for making code that interacts with type-destabilizing NA’s
>> faster. More generally, we need to make DataArrays and DataFrames fit in
>> better with Julia when Julia and R disagree. Following R’s lead has often
>> lead us astray because R doesn’t share Julia’s strenths or weaknesses.
>>
>> (4) Going forward, there should be exactly one way to do most things. The
>> worst part of our current codebase is that there are multiple ways to
>> express the same computation, but (a) some of them are unusably slow and (b)
>> some of them don’t ever get tested or maintained properly. This is closely
>> linked to the excess proliferation of functionality described in Resolution
>> 1 above. We need to start removing stuff from our packages and making the
>> parts we keep both reliable and fast.
>>
>> I think we can push DataArrays and DataFrames to 1.0 status by the end of
>> this year. But I think we need to adopt a new approach if we’re going to get
>> there. Lots of stuff needs to get deprecated and what remains needs a lot
>> more testing, benchmarking and documentation.
>>
>>  — John
>>
>

[julia-users] Re: Higher order derivatives in Calculus

[Hans W Borchers]

Think of a special function not (yet) implemented in Julia, like Lambert W, 
or Struve, or ... So you define it yourself, e.g. by applying 
Newton-Raphson. How would you symbolically differentiate that? Or your 
application forces you to define a procedure with if-else constructs and 
loops, how to differentiate? Even an advanced "automated differentiation" 
approach will most often not be capable of solving this for you.

Or imagine having to minimize a function such as

function repelling_balls(x::Array{Float64, 1})
  if length(x) != 45; error("Arg. 'x' must be a vector of length 45."); end
  d = 2.0
  for i = 1:14, j = (i+1):15
  s = sqrt((x[i]-x[j])^2 + (x[i+15]-x[j+15])^2 + (x[i+30]-x[j+30])^2)
  if s < d; d = s; end
  end
  return -d
end

that models 15 repelling balls/points in [0, 1]^3.

As "minimax" problem this function is smooth except for a subset of lower 
dimension. Some optimization procedures using gradients will be able to 
circumvent this subset and find a (local) minimum. How will you 
differentiate that (i.e., build a gradient) if not numerically; and if yes, 
is it worth the effort?

Consider that an optimization or root finding procedure is a (hopefully) 
converging process that will not return an absolutely precise result. A 
function itself may be defined through an optimization or root finding task 
and whose derivative will have a technical meaning.

In real-world applications, and tasks Matlab is made for, you do not care 
for 15 digits, and gradients five, six, seven digits accurate are more than 
enough to solve your problem. Normally, your functions are not defined for 
eps() accuracy anyway.

When I came here, I thought Julia was meant for "technical computing" and 
not so much for pure Math exercises. In scientific computing, calculating 
numerical derivatives and gradients is essential for many applications.


On Tuesday, January 21, 2014 3:28:28 AM UTC+1, Jason Merrill wrote:
>
> Would you be willing to share a problem that you're interested in that 
> isn't amenable to symbolic differentiation, maybe on a gist or something? 
> I've been looking for non-trivial problems to try to apply PowerSeries to.
>
> When I said numerical differentiation is "unstable", I meant something 
> like "hard to do without losing several digits". For example, you point out 
> that by choosing a different h, it's possible to improve over Calculus.jl's 
> current behavior by 2-3 decimal digits, from 7e-7 to 2e-9 in absolute 
> error. But the best precision you could hope for in a floating point answer 
> to this problem is
>
> julia> eps(sin(1.0))
> 1.1102230246251565e-16
>
> so even with your improvements, you've lost a lot of precision. That might 
> or might not be a problem depending on what you'd like to do next with the 
> value you get out.
>
>
> On Monday, January 20, 2014 12:30:38 PM UTC-8, Hans W Borchers wrote:
>>
>> Numerical differentiation is by far not as unstable as you seem to think.
>> And I have a long experience in using numerical derivatives for 
>> optimization problems where you don't stop to look up symbolic derivatives 
>> applying a CAS.
>> The function obviously was only an example.
>> For most of the functions I have used Julia's symbolic capabilities will 
>> not be sufficient.
>>
>>
>> On Monday, January 20, 2014 9:07:02 PM UTC+1, Jason Merrill wrote:
>>>
>>> This implementation could certainly use some love, but finite difference 
>>> differentiation is always unstable, and the situation gets worse as you 
>>> take higher order derivatives.
>>>
>>> You might want to consider using the differentiate method to take your 
>>> derivatives symbolically (if this works for the functions you're using), or 
>>> have a look at the differentiation example in PowerSeries.jl. To do a 
>>> symbolic second derivative, you can do, e.g.
>>>
>>> julia> using Calculus
>>> julia> @eval d2(x) = $(differentiate(differentiate(:(sin(x)
>>> julia> d2(1.0)
>>> -0.8414709848078965
>>>  
>>>
>>