[julia-users] Re: Using Interpolations.jl How to define the behavior when out of bounds?

[Tomas Lycken]

Sorry for not getting back to you sooner on this. 

Indexing into an extrapolation object like that should work - if you haven't 
already solved that problem, could you please file an issue with a complete 
(runnable) code sample? 

Regarding the comprehension, did you do that in the repl or in a script? Does 
it still yield Any[] if you wrap it in a function? If so, it might indicate a 
type stability issue in extrapolate - please file an issue for that too then. 

Thanks! 

// T 

[julia-users] Re: ANN: Julia 0.5.0-rc2 now available

[Tomas Lycken]

I just took the time to read the release notes 

 
more carefully, and I just want to congratulate everyone involved on a 
fantastic list of improvements. I've been away from the community for a 
little while, and expected 0.5 to be basically just the breaking array 
changes, but there's lots of goodies here.

Great job, everyone!

// T

On Friday, August 12, 2016 at 2:38:20 PM UTC+2, Tony Kelman wrote:
>
> I have just tagged and uploaded release candidate 2 for Julia version 
> 0.5.0. Binaries are available from
>
>
> https://s3.amazonaws.com/julialang/bin/linux/x64/0.5/julia-0.5.0-rc2-linux-x86_64.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/linux/x86/0.5/julia-0.5.0-rc2-linux-i686.tar.gz
>  
>
> https://s3.amazonaws.com/julialang/bin/osx/x64/0.5/julia-0.5.0-rc2-osx10.7+.dmg
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x64/0.5/julia-0.5.0-rc2-win64.exe
>  
>
> https://s3.amazonaws.com/julialang/bin/winnt/x86/0.5/julia-0.5.0-rc2-win32.exe
>  
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.sha256 
> https://s3.amazonaws.com/julialang/bin/checksums/julia-0.5.0-rc2.md5 
> For gpg signatures (with this key http://julialang.org/juliareleases.asc) 
> of the Linux tar.gz binaries, append .asc to the filename.
>
> (arm binaries are taking a while to build, I will upload them later - we 
> will also put links to this release candidate on the web site soon)
>
> The primary thing this does not yet include that we do plan on getting 
> into the final 0.5.0 release is proxy support for the package manager. A 
> preliminary version of that has been merged to master but still has some 
> build system issues that need to be worked out. We will put out a release 
> candidate 3 next week that will hopefully have this resolved, along with 
> any other major bug fixes that happen by then. If all goes well and no 
> major blocking issues come up after that, RC3 could possibly be the last 
> release candidate and promoted to final, but we will see how it goes next 
> week. Follow the progress at 
> https://github.com/JuliaLang/julia/issues/17418 and please report any 
> issues you find.
>
> -Tony
>
>

Re: [julia-users] Re: "eval cannot be used in a generated function"

[Tomas Lycken]

Julia’s parametric types are not generic enough to support this.

In my experience, any time I’ve been inclined to reason like this about one 
of my use cases, someone (usually Tim Holy :P) has been able to show me a 
way that works just fine, without magic, by just using the type system in a 
smarter way than I was able to figure out myself. So, if you’ll forgive my 
delusions of grandeur, I’ll attempt to suggest a way to support the cases 
you’ve mentioned so far!

The code turned out to be quite lengthy, so I put it in a gist instead: 
https://gist.github.com/tlycken/e501e1079d947d699b71941d93b7113e

Basically, what I’ve done is to create a generic type that holds all the 
information you need ((lower bound, stride, end) for each dimension) in its 
type arguments, and then a generator function that lets you specify these 
the same way you do today - with symbols as placeholders to be filled in 
when instantiated.

The generator function won’t be type stable, because the return type will 
depend on values of keyword arguments, but that shouldn’t be a big problem 
since it’s just upon array initialization; I suspect you will do the actual 
work in some kernel function, which then has access to the full type 
information.

// T

On Friday, August 12, 2016 at 7:38:07 AM UTC+2, Andy Ferris wrote:

> I'm encountering the error "eval cannot be used in a generated function" 
> in Julia 0.5 for code that is working in Julia 0.4. 
>
> I feel your pain, Erik!
>
> > AFAICT, it remains possible to do dynamic type generation if you (1) 
> print the code that would define the type to a file, and (2) `include` the 
> file. 
>
> While this is true, is your example meant to be a generated function? If I 
> need to recreate an existing type by calling `create_type_dynamically` 
> again with the same parameters, it won't be free at run-time. Unless we can 
> get it to recompile, and `isdefined()` is `@pure` or (evaluated by 
> inference)... hmm...
>
> > The definition of a generated function is "a pure function which 
> computes a function body based solely upon properties of the input argument 
> types". Since `eval` isn't pure, that would violate the definition of what 
> `@generated` is, and therefore it isn't usable. This isn't supposed to be 
> complicated, what an `@generated` function is actually supposed to be 
> allowed to do is just extremely limited, to make sure the system doesn't 
> fall apart on edge cases.
>
> Jameson - I suppose most of us think not of Julia as like the C or C++ 
> language standards, but as an implementation. A beautifully hackable 
> implementation, at that! I would assert that the definition of a generated 
> function is written in C++, lisp, Julia, etc. So it seemed to us users that 
> what "pure" meant was in this context was whatever we observed to work: 
> that it should always produce the same output code, and it seemed obvious 
> that the generator itself runs one or more times before the function is 
> called. Side effects that don't cause the compiler to get confused? Well, 
> why not? We can `println` in the generator, and that is rather useful for 
> debugging. In the past months several people have independently come up 
> with the same incredibly useful trick/hack, and there is now a bit of 
> sadness felt that our clever/insane code has died!
>
> I'm not saying this is terribly bad. Fixing #265 would be wonderful, and 
> we shouldn't get in the way of that. But Julia has always appeared to be 
> driven by community desires, and I feel these generated types are pretty 
> nifty. Constructive thoughts on any way forward would be greatly 
> appreciated. As would an actual example of this not working as expected in 
> the current implementation of Julia (though I think you may have provided 
> this for planned work on #265??).
>
​

Re: [julia-users] Re: "eval cannot be used in a generated function"

[Tomas Lycken]

Late to the party, but what’s wrong with writing FastArray{1,10,1,10} (or 
even FastArray{10,10} if you’re OK with implicit 1-based indexing)? It 
seems that those (valid) type arguments could convey just as much 
information as FastArray(1:10, 1:10), and you could then handle any 
special-casing on the size in the constructor. That way you’d only need one 
(generic) type. Or am I missing something important here?

// T

On Thursday, August 11, 2016 at 12:50:14 AM UTC+2, Erik Schnetter wrote:

The upshot of the discussions seems to be "it won't work in 0.5 because the 
> experts say so, and there are no plans to change that". So I'm going to 
> accept that statement.
>
> I think I'll use the following work-around:
> ```Julia
> immutable Wrapper{Tag,Types}
> data::Types
> end
> ```
> where I use `Tag` as `Val{Symbol}` to generate many distinct types, and 
> `Types` will be a tuple. This allows me to "generate" any immutable type. I 
> won't be able to access fields via the `.fieldname` syntax, but that is not 
> important to me.
>
> -erik
>
>
>
> On Wed, Aug 10, 2016 at 6:09 PM, Tim Holy  > wrote:
>
>> AFAICT, it remains possible to do dynamic type generation if you (1) 
>> print the
>> code that would define the type to a file, and (2) `include` the file.
>>
>> function create_type_dynamically{T}(::Type{T})
>> type_name = string("MyType", T)
>> isdefined(Main, Symbol(type_name)) && return nothing
>> filename = joinpath(tempdir(), string(T))
>> open(filename, "w") do io
>> println(io, """
>> type $type_name
>> val::$T
>> end
>> """)
>> end
>> eval(include(filename))
>> nothing
>> end
>>
>> Is this somehow less evil than doing it in a generated function?
>>
>> Best,
>> --Tim
>>
>> On Wednesday, August 10, 2016 9:49:23 PM CDT Jameson Nash wrote:
>> > > Why is it impossible to generate a new type at run time? I surely can 
>> do
>> >
>> > this by calling `eval` at module scope.
>> >
>> > module scope is compile time != runtime
>> >
>> > > Or I could create a type via a macro.
>> >
>> > Again, compile time != runtime
>> >
>> > > Given this, I can also call `eval` in a function, if I ensure the
>> >
>> > function is called only once.
>> >
>> > > Note that I've been doing this in Julia 0.4 without any (apparent)
>> >
>> > problems.
>> >
>> > Sure, I'm just here to tell you why it won't work that way in v0.5
>> >
>> > > I'm not defining thousands of types in my code. I define one type, and
>> >
>> > use it all over the place. However, each time my code runs (for days!), 
>> it
>> > defines a different type, chosen by a set of user parameters. I'm also 
>> not
>> > adding constraints to type parameters -- the type parameters are just 
>> `Int`
>> > values.
>> >
>> > Right, the basic tradeoff required here is that you just need to 
>> provide a
>> > convenient way for your user to declare the type at the toplevel that 
>> will
>> > be used for the run. For example, you can just JIT the code for the 
>> whole
>> > run at the beginning:
>> >
>> > function do_run()
>> >   return @eval begin
>> >  lots of function definitions
>> >  do_work()
>> >   end
>> > end
>> >
>> > On Wed, Aug 10, 2016 at 5:14 PM Erik Schnetter > > wrote:
>> > > On Wed, Aug 10, 2016 at 1:45 PM, Jameson > > wrote:
>> > >> AFAIK, defining an arbitrary new type at runtime is impossible, 
>> sorry. In
>> > >> v0.4 it was allowed, because we hoped that people understood not to 
>> try.
>> > >> See also https://github.com/JuliaLang/julia/issues/16806. Note that 
>> it
>> > >> is insufficient to "handle" the repeat calling via caching in a Dict 
>> or
>> > >> similar such mechanism. It must always compute the exact final output
>> > >> from
>> > >> the input values alone (e.g. it must truly be const pure).
>> > >
>> > > The generated function first calculates the name of the type, then 
>> checks
>> > > (`isdefined`) if this type is defined, and if so, returns it. 
>> Otherwise it
>> > > is defined and then returned. This corresponds to looking up the type 
>> via
>> > > `eval(typename)` (a symbol). I assume this is as pure as it gets.
>> > >
>> > > Why is it impossible to generate a new type at run time? I surely can 
>> do
>> > > this by calling `eval` at module scope. Or I could create a type via a
>> > > macro. Given this, I can also call `eval` in a function, if I ensure 
>> the
>> > > function is called only once. Note that I've been doing this in Julia 
>> 0.4
>> > > without any (apparent) problems.
>> > >
>> > > Being able to define types with arbitrary constraints in the type
>> > >
>> > >> parameters works OK for toy demos, but it's intentionally rather
>> > >> difficult
>> > >> since it causes performance issues at scale. Operations on Array are
>> > >> likely
>> > >> to be much faster (including the allocation) than on Tuple (due to 
>> the
>> > >> cost
>> > >> of *not* allocating) unless that Tuple is very 

Re: [julia-users] Adding tuples

[Tomas Lycken]

I guess this also depends on whether the benchmark (and the real scenario) 
involves doing this many times for the same N, or many times for different 
N. With the same N, it's probably going to be faster with Val{N}, but with 
many different N, using Val{N} is just going to make you pay the JIT fee 
for every new N instead of re-using an already compiled (but slightly less 
optimized) function.

// T

On Wednesday, August 10, 2016 at 10:06:18 PM UTC+2, Kristoffer Carlsson 
wrote:
>
> The Val{N} version will makes the compiler specialize the tuple creation 
> for that specific value of N. It should be significantly faster for small 
> lengths of the tuple and the return value should also be inferrable. 
>
> On Wednesday, August 10, 2016 at 8:59:45 PM UTC+2, Jeffrey Sarnoff wrote:
>>
>> almost certainly -- I use BenchmarkTools for this sort of timing, and can 
>> recommend it.
>>
>> On Wednesday, August 10, 2016 at 2:56:17 PM UTC-4, Bill Hart wrote:
>>>
>>> For me, the version using CartesianIndex is exactly the same speed as 
>>> the syntax with the Val, which in turn is faster than without Val.
>>>
>>> It probably depends a lot on the application and what the compiler can 
>>> handle.
>>>
>>> Bill.
>>>
>>> On 10 August 2016 at 20:45, Jeffrey Sarnoff  
>>> wrote:
>>>
 relative to the same thing without the Val


 On Wednesday, August 10, 2016 at 2:45:06 PM UTC-4, Jeffrey Sarnoff 
 wrote:
>
> that slows it down by a factor of 5
>
> On Wednesday, August 10, 2016 at 2:37:25 PM UTC-4, Bill Hart wrote:
>>
>> How about compared with:
>>
>> ntuple(i -> a[i] + b[i], Val{N})
>>
>>
>> On 10 August 2016 at 20:32, Jeffrey Sarnoff  
>> wrote:
>>
>>> Bill,
>>>
>>> Following Eric's note, I tried (with a,b equi-length tuples)
>>> function addTuples(a,b)
>>> ca = CartesianIndex(a)
>>> cb = CartesianIndex(b)
>>> return (ca+cb).I
>>> end
>>>
>>>
>>> for me, with 100 values it ran ~60% faster, and with 1000 values 
>>> much much faster than  
>>>  ntuple(i -> a[i] + b[i], N)
>>>
>>>
>>>
>>> On Wednesday, August 10, 2016 at 11:06:46 AM UTC-4, Bill Hart wrote:

 This code seems to be (about 50%) faster than recursive functions:

 Base.:+{N}(a::NTuple{N}, b::NTuple{N}) = ntuple(i -> a[i] + b[i], N
> )


 But this seems (about 50%) slower:

  ((a[i] + b[i] for i = 1:N)...)


 Anyway, I can use the first method, until I find something faster. 
 It's definitely way more convenient. Thanks.

 Bill. 



 On 10 August 2016 at 16:56, Erik Schnetter  
 wrote:

> The built-in type `CartesianIndex` supports adding and 
> subtracting, and presumably also multiplication. It is implemented 
> very 
> efficiently, based on tuples.
>
> Otherwise, to generate efficient code, you might have to make use 
> of "generated functions". These are similar to macros, but they know 
> about 
> the types upon which they act, and thus know the value of `N`. This 
> is a 
> bit low-level, so I'd use this only if (a) there is not other package 
> available, and (b) you have examined Julia's performance and found it 
> lacking.
>
> I would avoid overloading operators for `NTuple`, and instead us a 
> new immutable type, since overloading operations for Julia's tuples 
> can 
> have unintended side effects.
>
> -erik
>
>
> On Wed, Aug 10, 2016 at 9:57 AM, 'Bill Hart' via julia-users <
> julia...@googlegroups.com> wrote:
>
>> Does anyone know an efficient way to add NTuples in Julia?
>>
>> I can do it using recursive functions, but for various reasons 
>> this is not efficient in my context. I really miss something like 
>> tuple(a[i] + b[i] for i in 1:N) to create the resulting tuple all in 
>> one go 
>> (here a and b would be tuples).
>>
>> The compiler doesn't do badly with recursive functions for 
>> handling tuples in very straightforward situations, but for example, 
>> if I 
>> want to create an immutable type based on a tuple the compiler 
>> doesn't seem 
>> to be able to handle the necessary optimisations. At least, that is 
>> what I 
>> infer from the timings. Consider
>>
>> immutable bill{N}
>>d::NTuple{N, Int}
>> end
>>
>> and I want to add two bill's together. If I have to add the 
>> tuples themselves using recursive functions, then I no longer seem 
>> to be 
>> 

[julia-users] Re: Why is Julia 0.5 built from source almost twice as large (on disk) as Julia 0.4?

[Tomas Lycken]

I'm now down to

2,2G  ./julia-0.4
1,7G  ./julia-0.5

which is good enough to stop the low disk space warnings :D

I'll be eagerly awaiting that PR, Jameson! ;) Thanks, all!

// T


On Wednesday, August 10, 2016 at 6:19:23 PM UTC+2, Jameson wrote:
>
> Yes, you can delete anything old (the same goes for usr-staging and 
> deps/srccache). I've also been slowly developing a PR that will allow the 
> build system to automatically erase the build directories after its 
> finished with them, but it's not ready yet. But someday...
>
>
>
> On Wednesday, August 10, 2016 at 5:33:50 AM UTC-4, Tomas Lycken wrote:
>>
>> Thanks for the replies.
>>
>> Is it safe to assume that anything in deps/build that exists in multiple 
>> versions, is only needed in the latest of those? For instance, I have 
>>
>> ```
>> 159M deps/build/llvm-3.3
>> 318M deps/build/llvm-3.7.1
>> 881M deps/build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6
>> 943M deps/build/openblas
>> ```
>>
>> where it seems I could shave off a GB or so by deleting `llvm-3.3` and 
>> `openblas-`. `deps/srccache` is another 650 MB, can that also be 
>> deleted?
>>
>> My main reason for building from source rather than using the binaries is 
>> that now and then I stumble on something I want to investigate and/or 
>> improve in base, and the threshold for actually filing a PR is much lower 
>> if I already have the code I'm running locally. Once 0.5 is out for real 
>> I'll probably drop the source tree for 0.4, so "temporarily" freeing up a 
>> couple of gigs by deleting build intermediates is good enough for now.
>>
>> // T
>>
>> On Wednesday, August 10, 2016 at 10:49:38 AM UTC+2, Andreas Lobinger 
>> wrote:
>>>
>>> Hello colleague,
>>>
>>> On Wednesday, August 10, 2016 at 10:11:46 AM UTC+2, Tomas Lycken wrote:
>>>>
>>>> Both instances of Julia are runnable, so I don’t think I deleted 
>>>> something I shouldn’t have in either folder. 
>>>>
>>>> What has changed to make Julia 0.5 so big? Are there any build 
>>>> artifacts I can/should prune to reduce this footprint?
>>>>
>>> my guess it's some cumulative build artefacts:
>>>
>>>  lobi@orange4:~/julia05/deps$ du -sh *
>>> 8,0Karpack.mk
>>> 12K blas.mk
>>> 4,8Gbuild
>>> 508Kchecksums
>>> 4,0Kdsfmt.mk
>>> 8,0Kfftw.mk
>>> 4,0Kgfortblas.alias
>>> 8,0Kgfortblas.c
>>> 4,0Kgmp.mk
>>> 4,0KlibdSFMT.def
>>> 4,0Klibgit2.mk
>>> 4,0Klibgit2.version
>>> 4,0Klibssh2.mk
>>> 4,0Klibssh2.version
>>> 4,0Klibuv.mk
>>> 4,0Klibuv.version
>>> 20K llvm.mk
>>> 4,0Kllvm-ver.make
>>> 8,0KMakefile
>>> 4,0Kmbedtls.mk
>>> 4,0Kmpfr.mk
>>> 4,0KNATIVE.cmake
>>> 4,0Kobjconv.mk
>>> 4,0Kopenblas.version
>>> 4,0Kopenlibm.mk
>>> 4,0Kopenlibm.version
>>> 4,0Kopenspecfun.mk
>>> 4,0Kopenspecfun.version
>>> 4,0Kpatchelf.mk
>>> 328Kpatches
>>> 4,0Kpcre.mk
>>> 2,0Gsrccache
>>> 8,0Ksuitesparse.mk
>>> 4,0KSuiteSparse_wrapper.c
>>> 20K tools
>>> 4,0Kunwind.mk
>>> 4,0Kutf8proc.mk
>>> 4,0Kutf8proc.version
>>> 384Kvalgrind
>>> 4,0KVersions.make
>>> 4,0Kvirtualenv.mk
>>>
>>>

Re: [julia-users] Re: map(mean, zip(v...)) doesn't scale

[Tomas Lycken]

I don't think there are such binaries - you'd have to build from source - 
but my reproduction was on rc1+1, so that bug is (probably) not relevant to 
this issue.

// T

On Wednesday, August 10, 2016 at 1:12:57 PM UTC+2, Tamas Papp wrote:
>
> Where can I find (64bit, Linux) binaries for rc1+1? 
>
> On Wed, Aug 10 2016, Andreas Lobinger wrote: 
>
> > There is a memory allocation(?) bug in 0.5.0-rc1+0, recommendation is to 
> go 
> > to rc1+1. 
> > 
> > On Wednesday, August 10, 2016 at 12:04:36 PM UTC+2, Tamas Papp wrote: 
> >> 
> >> Thanks. Forgot to say that I am using v"0.5.0-rc1+0". 
> >> 
> >> 
>
>

[julia-users] Re: map(mean, zip(v...)) doesn't scale

[Tomas Lycken]

For completeness: I can reproduce this behavior for m1. m2 returns fine 
after a little over four times the time it took for 100 vectors.

// T

On Wednesday, August 10, 2016 at 11:34:08 AM UTC+2, Tamas Papp wrote:
>
> I was benchmarking code for calculating means of vectors, eg 
>
> v = [rand(100) for i in 1:100] 
>
> m1(v) = map(mean, zip(v...)) 
> m2(v) = mapslices(mean, hcat(v...), 2) 
>
> @elapsed m1(v) 
> @elapsed m2(v) 
>
> m2 is faster, but for 1000 vectors, 
>
> v = [rand(100) for i in 1:1000] 
>
> m1 with zip takes "forever" (CPU keeps running at 100%, I interrupted 
> after a 10 minutes). Is this supposed to happen, or should I report it 
> as a bug? 
>
> Tamas 
>

[julia-users] Re: Why is Julia 0.5 built from source almost twice as large (on disk) as Julia 0.4?

[Tomas Lycken]

Thanks for the replies.

Is it safe to assume that anything in deps/build that exists in multiple 
versions, is only needed in the latest of those? For instance, I have 

```
159M deps/build/llvm-3.3
318M deps/build/llvm-3.7.1
881M deps/build/openblas-12ab1804b6ebcd38b26960d65d254314d8bc33d6
943M deps/build/openblas
```

where it seems I could shave off a GB or so by deleting `llvm-3.3` and 
`openblas-`. `deps/srccache` is another 650 MB, can that also be 
deleted?

My main reason for building from source rather than using the binaries is 
that now and then I stumble on something I want to investigate and/or 
improve in base, and the threshold for actually filing a PR is much lower 
if I already have the code I'm running locally. Once 0.5 is out for real 
I'll probably drop the source tree for 0.4, so "temporarily" freeing up a 
couple of gigs by deleting build intermediates is good enough for now.

// T

On Wednesday, August 10, 2016 at 10:49:38 AM UTC+2, Andreas Lobinger wrote:
>
> Hello colleague,
>
> On Wednesday, August 10, 2016 at 10:11:46 AM UTC+2, Tomas Lycken wrote:
>>
>> Both instances of Julia are runnable, so I don’t think I deleted 
>> something I shouldn’t have in either folder. 
>>
>> What has changed to make Julia 0.5 so big? Are there any build artifacts 
>> I can/should prune to reduce this footprint?
>>
> my guess it's some cumulative build artefacts:
>
>  lobi@orange4:~/julia05/deps$ du -sh *
> 8,0Karpack.mk
> 12K blas.mk
> 4,8Gbuild
> 508Kchecksums
> 4,0Kdsfmt.mk
> 8,0Kfftw.mk
> 4,0Kgfortblas.alias
> 8,0Kgfortblas.c
> 4,0Kgmp.mk
> 4,0KlibdSFMT.def
> 4,0Klibgit2.mk
> 4,0Klibgit2.version
> 4,0Klibssh2.mk
> 4,0Klibssh2.version
> 4,0Klibuv.mk
> 4,0Klibuv.version
> 20K llvm.mk
> 4,0Kllvm-ver.make
> 8,0KMakefile
> 4,0Kmbedtls.mk
> 4,0Kmpfr.mk
> 4,0KNATIVE.cmake
> 4,0Kobjconv.mk
> 4,0Kopenblas.version
> 4,0Kopenlibm.mk
> 4,0Kopenlibm.version
> 4,0Kopenspecfun.mk
> 4,0Kopenspecfun.version
> 4,0Kpatchelf.mk
> 328Kpatches
> 4,0Kpcre.mk
> 2,0Gsrccache
> 8,0Ksuitesparse.mk
> 4,0KSuiteSparse_wrapper.c
> 20K tools
> 4,0Kunwind.mk
> 4,0Kutf8proc.mk
> 4,0Kutf8proc.version
> 384Kvalgrind
> 4,0KVersions.make
> 4,0Kvirtualenv.mk
>
>

[julia-users] Why is Julia 0.5 built from source almost twice as large (on disk) as Julia 0.4?

[Tomas Lycken]

After being away from Julia and the community for a couple of months, I 
started updating my Julia installations today in order to test some of my 
packages and make sure they’re ready for 0.5, and I started getting 
warnings about disk usage. Since I’m dual-booting Ubuntu and Windows on a 
laptop with only about 120GB SSD space, disk space is precious and every GB 
counts.

I have two parallel installations of Julia on my laptop, both built from 
source - the release-0.4 branch and the release-0.5 branch (this folder 
used to contain the master branch, but I checked out release-0.5 today). 
After building with make (preceded by make cleanall in the 0.5 folder) and 
pruning everything indicated by git status to be added (mostly tarballs and 
a couple of repos inside deps/) I now have the following status:

/opt/julia-0.4  release-0.4 $ git status
On branch release-0.4
Your branch is up-to-date with 'origin/release-0.4'.
nothing to commit, working directory clean

/opt/julia-0.5  release-0.5 $ git status
On branch release-0.5
Your branch is up-to-date with 'origin/release-0.5'.
nothing to commit, working directory clean

/opt $ du -hd1 | sort -hr
6,8G.
4,0G./julia-0.5
2,3G./julia-0.4
# smaller stuff omitted

Both instances of Julia are runnable, so I don’t think I deleted something 
I shouldn’t have in either folder.

What has changed to make Julia 0.5 so big? Are there any build artifacts I 
can/should prune to reduce this footprint?

// T
​

Re: [julia-users] Creating symbols

[Tomas Lycken]

I am all for changing this, but in the specific case of symbol/Symbol this is 
going to be massively breaking, and even if the fix is pretty simple (applying 
s/symbol\(/Symbol\(/ probably fixes 99% of the code) the timing needs to be 
right. 

What other cases are there of such functions that should be merged into 
constructors? Is there an issue to track them? 

Also, when is a good time to introduce such a change?  Should they be tackled 
at the same time, or at a different one? 

I'm willing to pitch in some time to help here, if the timing works out.

// T 

[julia-users] Re: Warning on operations mixing BigFloat and Float64

[Tomas Lycken]

Adding a BigFloat and a Float64 should automatically promote both to 
BigFloats, avoiding precision loss for you.

julia> BigFloat(2.9) + 0.3
3.199900079927783735911361873149871826171875

Do you have a case where this doesn’t happen?

// T

On Monday, April 18, 2016 at 4:32:52 PM UTC+2, Paweł Biernat wrote:

Hi,
>
> I want to make sure I am not loosing any precision in my code by 
> accidentally mixing BigFloat and Float64 (e.g. adding two numbers of 
> different precision).  I was thinking about replacing the definitions of 
> `+`, `-`, etc. for BigFloat but if you do that for all two argument 
> functions this would be a lot of redefining, so I started to wonder if 
> there is a more clever approach.  Is there any simple hack to get a warning 
> if this happens?
>
> Best,
> Paweł
>
> ​

Re: [julia-users] 'do' notation in documentation?

[Tomas Lycken]

If you'd like to share that document, it would be a valuable resource for 
others as well when trying to help out. It's not unlikely that at least a 
subset of the things in your list are things that are present (and perhaps 
even quite extensively so) in the documentation, but explained in a way 
that's more suitable for an experienced Julia programmer than for someone 
who has never heard of the concepts before. Such documentation issues are 
very hard to notice for the people who might be best suited to try to fix 
them, so even if you don't have time to adress them yourself, just sharing 
the list will make it easier for others to help mitigate it.

// T

On Sunday, April 17, 2016 at 10:13:23 PM UTC+2, Andrew Gibb wrote:
>
> I have a little document which I add to periodically called "Julia 
> unsearchables" where I note down which things can't be found in the docs 
> without asking someone. I do hope, once my Thesis is done, that I can 
> contribute it to the documentation. 

[julia-users] How to initialize a Matrix{T}?

[Tomas Lycken]

Matrices in Julia, unlike some other languages, aren't vectors of vectors - 
they are first class citizens of the multidimensional array system in Julia. 
Thus, 

A = Array(Float64,1,1)

creates a 1x1 matrix that holds Float64s. In most cases, it's even better to 
use 

A = zeros(Float64, 1, 1)

to ensure that you don't have any garbage data (Array allocates, but doesn't 
wipe the memory). Whether that is possible in your case depends on the type 
parameter data. 

// T 

[julia-users] strategy design pattern

[Tomas Lycken]

Multiple dispatch is actually really well suited for this. The idiomatic 
approach in Julia is quite similar to your c++ example, at least conceptually:

immutable Foo end
immutable Bar end

alg(::Type{Foo}) = println("I'm using foo") 
alg(::Type{Bar}) = println("I'm using bar") 

strategy = Foo # or Bar 
alg(strategy) 

// T 

Re: [julia-users] Re: Julia is a great idea, but it disqualifies itself for a big portion of its potential consumers

[Tomas Lycken]

There are only 6 uses of // outside of rationals.jl in base (4 of those are 
in mpfr.jl), compared to 187 uses of div, and 22 uses of ÷. (that’s uses, 
not definitions, exports, documentation, although the ratios are very 
similar).
Looking at packages, it seems also that div is used frequently, and many 
times more than // for rational numbers.

I don’t think counting usages of // vs ÷ is necessarily fair, since I think 
// is used less than it should be (for reasons I eventually expanded into a 
section in the Julia style guide 
).
 
More fair would be to compare the number of usages of ÷ with the number of 
usages of // *plus the number of float literals*. (The results might be the 
same, though - I haven’t actually made the comparison).

But more to the point:

That is your opinion, however a number of other people disagree.

*Exactly.* People disagree on this, and there seems to be a shortage of 
ASCII characters (or character sequences) for everyone to have it all. In 
the table you posted a while ago, I notice that Julia is the *only* 
language which has infix operators for both (int, int) -> float, (int, int) 
-> int and (int, int) -> rational division. I’m not saying the design of 
this is final (software design is never final for software reasons - only 
for people reasons…) but to me it looks like Julia has already hit a sweet 
spot in supporting as much as possible for as many tastes as possible.

Note that what the author *really wants* is to get the i-th potion of an 
array

Tamas Papp already posted an idiomatic solution for this:

Define 

portion(i,n,m=100) = 1+div(i*m,n):div((i+1)*m,n)

and use 

a[portion(i,n)]

// T

On Tuesday, April 5, 2016 at 4:55:05 AM UTC+2, Scott Jones wrote:


>
> On Monday, April 4, 2016 at 7:27:40 AM UTC-4, Stefan Karpinski wrote:
>>
>> Number does not imply or assume commutativity. The Quaternions package 
>> provides a Quaternion type which is a subtype of Number. Rational, however, 
>> only allows integer numerators and denominators. Since integer 
>> multiplication is commutative, rational multiplication is too. But I still 
>> think it best to reserve \\ with an analogous meaning to //. There are 
>> already two syntaxes for integer division, which is plenty.
>>
>
> That is your opinion, however a number of other people disagree. Every 
> other language I've dealt with has had a simple ASCII sequence for integer 
> division, / (in C, C++, Java, Python2 with integer operands), \ (in Mumps & 
> Caché Object Script), or // (Python and Lua).
> (and no, typing \div in the REPL and something else in an editor [as 
> well as having to customize one's editor] just to get a Unicode ÷ 
> character, is not really that useful)
>
> About \\ in particular, that's fine then, probably better to reserve that 
> for rationals.
> To me, it does seem a bit strange though that // was picked for rationals 
> in Julia, something sure to cause confusion for people coming from Python 
> (which seems to be a large part of people moving to Julia) or Lua, when 
> integer division is used so much more frequently than rational numbers.
> There are only 6 uses of // outside of rationals.jl in base (4 of those 
> are in mpfr.jl), compared to 187 uses of div, and 22 uses of ÷.  (that's 
> uses, not definitions, exports, documentation, although the ratios are very 
> similar).
> Looking at packages, it seems also that div is used frequently, and many 
> times more than // for rational numbers.
>
> What about a different two character ASCII sequence that is currently a 
> syntax error?
> I think /| could be used, and could be easily remembered as being integer 
> division (unlike ÷), it could be described as doing a division / and 
> cutting off the remainder |.
>
​

[julia-users] Re: enforcing homogeneity of vector elements in function signature

[Tomas Lycken]

The reason you’re getting that method error is because of type parameter 
invariance 

 
- in short, even though F <: Foo{T}, we *don’t* have Vector{F} <: 
Vector{Foo{T}}. Until triangular dispatch lands, defining two methods is 
probably the best you can do (but if the method definition is long enough 
to warrant it, you could separate the actual implementation out into a 
different helper function, e.g. _bar(x::Vector) which has too-relaxed types 
on its parameters, but that shouldn’t be a problem since it’s only internal.

Another option is to explicitly make sure that the vector is typed to be 
homogeneous only in the first argument, although all the elements are 
actually homogeneous in both:

function make_homogeneous_Foos(M)
fs = Foo{Float64}[]
...
end

// T

On Monday, April 4, 2016 at 10:46:21 PM UTC+2, Davide Lasagna wrote:

Thanks, yes, I have tried this, but did not mention what happens.
>
> For the signature you suggest, you get a `MethodError` in the case the 
> vector `x` is homogeneous in both parameters.
>
> Look at this code
>
> type Foo{T, N}
> a::NTuple{N, T}
> end
>
> function make_homogeneous_Foos(M)
> fs = Foo{Float64, 2}[]
> for i = 1:M
> f = Foo{Float64, 2}((0.0, 0.0))
> push!(fs, f)
> end
> fs
> end
>
> function bar{T}(x::Vector{Foo{T}})
> println("Hello, Foo!")
> end
>
> const fs = make_homogeneous_Foos(100)
>
> bar(fs)
>
> which results in 
>
> ERROR: LoadError: MethodError: `bar` has no method matching 
> bar(::Array{Foo{Float64,2},1})
>
> A workaround would be to have two methods, one for the homogeneous 
> elements in the first parameter, as you suggest, and a second for a vector 
> with homogeneous elements in both parameters, with both T, N specified in 
> the signature. But I have to write an extra method...
>
>
> On Monday, April 4, 2016 at 9:32:55 PM UTC+1, John Myles White wrote:
>>
>> Vector{Foo{T}}?
>>
>> On Monday, April 4, 2016 at 1:25:46 PM UTC-7, Davide Lasagna wrote:
>>>
>>> Hi all, 
>>>
>>> Consider the following example code
>>>
>>> type Foo{T, N}
>>> a::NTuple{N, T}
>>> end
>>>
>>> function make_Foos(M)
>>> fs = Foo{Float64}[]
>>> for i = 1:M
>>> N = rand(1:2)
>>> f = Foo{Float64, N}(ntuple(i->0.0, N))
>>> push!(fs, f)
>>> end
>>> fs
>>> end
>>>
>>> function bar{F<:Foo}(x::Vector{F})
>>> println("Hello, Foo!")
>>> end
>>>
>>> const fs = make_Foos(100)
>>>
>>> bar(fs)
>>>
>>> What would be the signature of `bar` to enforce that all the entries of 
>>> `x` have the same value for the first parameter T? As it is now, `x` could 
>>> contain an `Foo{Float64}` and a `Foo{Int64}`, whereas I would like to 
>>> enforce homogeneity of the vector elements in the first parameter.
>>>
>>> Thanks
>>>
>>>
>>> ​

Re: [julia-users] Re: Julia is a great idea, but it disqualifies itself for a big portion of its potential consumers

[Tomas Lycken]

It’s also entirely possible to create a macro that lets / do what you (the 
OP) want.

@integer_division begin

# in here, 5 / 2 == 2

end

Basically, you’d just traverse the AST recursively and switch all calls to / 
for calls to div. (I admit, though I wouldn’t want to do it myself, nor 
would I want to maintain code that uses it, but at least the latter is 
probably just a matter of taste…)

The fact that this is even possible in Julia, is in my eyes a perfect 
counter-argument to most of the pain points in the original post. I know of 
no other language which is so painless to use when conforming to the 
(usually very good) design decisions that have already been made, while 
simultaneously giving the user the power to override almost anything if 
desired.

Similarly, it has already been brought-up that it’s quite possible to 
create 0-indexed arrays (and making that even easier is even a current 
hot-topic).

As others have already brought up, there are lots of people who feel quite 
opposite about both integer division and 0- or 1-based indexing. Satisfying 
everyone isn’t going to be possible, but Julia comes damn close.

// T

On Monday, April 4, 2016 at 12:55:57 PM UTC+2, Tim Holy wrote:

On Sunday, April 03, 2016 09:28:28 AM Scott Jones wrote: 
> > With \\, would you worry about confusion from the fact that in a \\ b, a 
> is 
> > 
> > > in 
> > > the denominator? Especially if it gets called the "integer division 
> > > operator." 
> > 
> > It might confuse some of the mathematicians, used to a \ b as a inverse 
> > multiplication operator, however, other languages use a \ b for what in 
> > Julia is div(a, b) or a÷b, 
> > so I really don't think a definition of \\ as div(a, b) would be that 
> much 
> > of a problem, no worse than other things you have to remember in Julia, 
> > and it's reminiscent of the // integer division operator in Python and 
> Lua. 
>
> Thinking about it a second time, there's another reason I'd recommend 
> against 
> this choice: if you're working with non-commutative numbers, for 
> consistency 
> \\ too should be reserved for forming Rationals rather than meaning 
> something 
> completely different. In other words, b//a means RightRational(b, a) 
> (b*inv(a)) 
> and a\\b means LeftRational(a, b) (inv(a)*b). I suspect our current 
> Rational 
> framework implicitly assumes commutativity, but that's a separate issue 
> and 
> one that, with a suitable number-traits system, is cleanly resolvable. 
>
> I'm not trying to argue that making rationals with non-commutative numbers 
> is 
> a daily task, but to me consistency in the meaning of notation seems like 
> the 
> most important guiding principle. Code gets read many more times than it 
> is 
> written, so I don't mind a few extra keystrokes if it makes reading code 
> clearer. 
>
> But once again, it seems there just aren't enough characters on the 
> keyboard. 
>
> Best, 
> --Tim 
>
> ​

[julia-users] Re: How to specify a template argument as a super type

[Tomas Lycken]

If it's abstract, what is the purpose of the type parameter? In other 
words, what is the ultimate goal you're reaching for?

//T 

On Saturday, April 2, 2016 at 3:17:34 AM UTC+2, Scott Lundberg wrote:
>
> You can define:
>
> abstract Name1{S <: Name2} <: Name3{S}
>
> but I want to define:
>
> abstract Name1{S <: Name2} <: S
>
> However the second line does not work. Is there some other way to 
> accomplish the same thing? Thanks!
>

Re: [julia-users] Announcing JuDE: autocomplete and jump to definition support for Atom

[Tomas Lycken]

It would be nice, though, if it were somewhere documented - or even better, 
a meta-package with dependencies on all the others - to allow turning an 
already existing Atom installation into a Julia IDE *too*. I'd be much more 
interested in improving an editor I'm already using for other thing, than 
in installing yet another atom-sublime-vscode-like (they all look the same 
nowadays...) editor just to get the Julia goodness.

This is, of course and as already noted, definitely not urgent, but I think 
it should be on the road map.

// T

On Monday, March 28, 2016 at 10:26:06 PM UTC+2, Jonathan Malmaud wrote:
>
> Mike can step in, but I think it is indeed the plan to have a Julia IDE 
> bundle, causing the exact packages required on the julia and atom side to 
> be invisible implementation details from the user's POV.
>
> On Monday, March 28, 2016 at 1:27:01 PM UTC-4, David Anthoff wrote:
>>
>> I think the user experience for almost all people would be much easier if 
>> they install Atom, then add one package for julia support and everything 
>> works. Having multiple Atom packages (julia-language, ink, julia-client, 
>> latex-completion and Jude right now) makes things confusing for most 
>> people, installation a pain and most people probably won’t even discover 
>> all of these packages.
>>
>>  
>>
>> Maybe another solution to this would be to eventually have one “juno” 
>> package that automatically installs all the individual useful plugins for 
>> Atom that make it a good julia IDE. I also don’t think any of this is 
>> urgent, right now the whole Atom story is so experimental in any case that 
>> most likely only a small number of people is using it in the first place.
>>
>>  
>>
>> *From:* julia...@googlegroups.com [mailto:julia...@googlegroups.com] *On 
>> Behalf Of *James Dang
>> *Sent:* Sunday, March 27, 2016 4:57 AM
>> *To:* julia-users 
>> *Subject:* Re: [julia-users] Announcing JuDE: autocomplete and jump to 
>> definition support for Atom
>>
>>  
>>
>> Hi David, I think Jude's functionality is pretty orthogonal to 
>> julia-client so they function pretty well as distinct, easier to manage 
>> packages. julia-client is focused on live code execution and results 
>> visualization while Jude is more about getting traditional IDE tools via 
>> static syntax analysis. Why do you think they should merge?
>>
>

[julia-users] Re: Shuffle matrix in julia

[Tomas Lycken]

What do you want to achieve - shuffling along a dimension (i.e. shuffling 
rows or columns), or just shuffling all the numbers?

For the latter, it can be achieved pretty easily by reshaping to a vector 
and back. This has the added benefit that it generalizes well to arbitrary 
dimensions:

julia> A = reshape(1:40, 4,5,2)
4x5x2 Array{Int64,3}:
[:, :, 1] =
 1  5   9  13  17
 2  6  10  14  18
 3  7  11  15  19
 4  8  12  16  20

[:, :, 2] =
 21  25  29  33  37
 22  26  30  34  38
 23  27  31  35  39
 24  28  32  36  40

julia> reshape(shuffle(vec(A)), size(A)...)
4x5x2 Array{Int64,3}:
[:, :, 1] =
 32  38   8  21   6
 29  25  31  13  20
 34  10   4  33  39
 18   5  27   7  15

[:, :, 2] =
 40  22  30  19  16
 11  23  14  36  26
 17   3  24   1  37
 35  28   9   2  12

// T

On Thursday, March 24, 2016 at 2:09:36 PM UTC+1, Arundhyoti Sarkar wrote:


>
> Is there a way to shuffle matrix in julia like we do in matlab?
> I know shuffle(v) shuffles the vector v of type Array{Any,1}. But doesnot 
> work with matrices. 
>
​

[julia-users] Re: npm, left_pad and julia

[Tomas Lycken]

I guess there's an interesting discussion lurking here too, on what license 
requirements should be applied to all packages that we register in 
METADATA. One of the reasons that npm could just restore left-pad to the 
package repositories without consent from the author, is that the code was 
released under the WTFPL . With a 
restrictive or proprietary license (e.g. "you may use this code, but not 
redistribute it"), I guess npm's laywers would have had a little more to 
think about.

According to the readme on the METADATA repository, a package must be 
licensed under an Open Source Initiative  approved 
license, which I guess covers our bases here as long as that policy is also 
enforced.

I assume that there are other package manager vendors that are reading the 
fine prints in all their EULA's now, though. NuGet.org (the largest package 
host for .NET libraries), for example, hosts loads of packages with just 
binaries - no source code - and there is no (enforced) licensing policy 
there.

// T

On Thursday, March 24, 2016 at 12:21:32 PM UTC+1, Tony Kelman wrote:
>
> We have a JuliaPackageMirrors organization that automatically mirrors git 
> repositories of registered packages. If someone deleted their repositories 
> on github, we can adjust the metadata url to point to the mirror.

[julia-users] Re: How do I, as efficiently as possible, iterate over the first and last elements along a given dimension of an array?

[Tomas Lycken]

Fantastic, thank you! Except for what I assume is a typo in `dropdim` (we 
want `splice!(v,i)`), this does exactly what I want.

// T

On Thursday, March 24, 2016 at 2:37:12 AM UTC+1, Dan wrote:
>
> Sorry for the bad formatting (getting tangled with the editor).
>
> Additional useful code to supplement the previous note:
>
> function dropdim(A,i)
>   v = collect(size(A))
>   splice!(A,i)
>   return tuple(v...)
> end
>
> # now something like this works:
> # which gives the sum of the two edge planes of array A3 as a matrix 
> reshape(map(x->A3[x[1]]+A3[x[2]],mapedges(A3,3)),dropdim(A3,3))
>
>
>
>
>
>
> On Thursday, March 24, 2016 at 3:21:02 AM UTC+2, Dan wrote:
>>
>> # something along these lines:
>>
>> topleft(A,d) = tuple(ones(Int,ndims(A))...)
>> bottomleft(A,d) = tuple([i==d ? 1 : e for (i,e) in enumerate(size(A
>> ))]...)
>>
>> topright(A,d) = tuple([i==d ? e : 1 for (i,e) in enumerate(size(A))]...)
>> bottomright(A,d) = size(A)
>> mapedges(A,d) = zip(
>>   CartesianRange{CartesianIndex{ndims(A)}}(
>> CartesianIndex{ndims(A)}(topleft(A,d)...),
>> CartesianIndex{ndims(A)}(bottomleft(A,d)...)),
>>   CartesianRange{CartesianIndex{ndims(A)}}(
>> CartesianIndex{ndims(A)}(topright(A,d)...),
>> CartesianIndex{ndims(A)}(bottomright(A,d)...)
>>   ))
>>
>> A3 = rand(10,15,8)
>>
>> julia> mapedges(A3,1) |> length
>> 120
>>
>> julia> mapedges(A3,2) |> length
>> 80
>>
>> julia> mapedges(A3,3) |> length
>> 150
>>
>>
>> On Thursday, March 24, 2016 at 2:53:48 AM UTC+2, Tomas Lycken wrote:
>>>
>>> …but not really. Reading the docstring more carefully:
>>>
>>> Transform the given dimensions of array A using function f. f is called 
>>> on each slice of A of the form A[…,:,…,:,…]. dims is an integer vector 
>>> specifying where the colons go in this expression. The results are 
>>> concatenated along the remaining dimensions. For example, if dims is [1,2] 
>>> and A is 4-dimensional, f is called on A[:,:,i,j] for all i and j.
>>>
>>> What I want to do, is rather call f on A[:,:,1,:] and A[:,:,end,:], but 
>>> nothing in between 1 and end for that dimension. mapslices still 
>>> eventually visit the entire array (either by slicing, or by iteration), but 
>>> I only want to visit the “edges”. I might be missing something, though.
>>>
>>> // T
>>>
>>> On Thursday, March 24, 2016 at 1:48:36 AM UTC+1, Tomas Lycken wrote:
>>>
>>> Yes, probably - thanks for the tip! I'll see if I can cook something 
>>>> up...
>>>>
>>>> On Thursday, March 24, 2016 at 1:45:32 AM UTC+1, Benjamin Deonovic 
>>>> wrote:
>>>>>
>>>>> Can mapslices help here?
>>>>>
>>>>>
>>>>> On Wednesday, March 23, 2016 at 6:59:59 PM UTC-5, Tomas Lycken wrote:
>>>>>>
>>>>>> Is there an effective pattern to iterate over the “endpoints” of an 
>>>>>> array along a given dimension?
>>>>>>
>>>>>> What I eventually want to accomplish is to apply a function (in this 
>>>>>> case an equality test) to the two end points along a particular 
>>>>>> dimension 
>>>>>> of an array. I think the pattern is easiest explained by considering 1D, 
>>>>>> 2D 
>>>>>> and 3D:
>>>>>>
>>>>>> # assume the existence of some scalar-valued function f(x,y)
>>>>>>
>>>>>> A1 = rand(10)
>>>>>> f(A1[1], A1[end]) # d == 1 (the only possible value) -> one evaluation
>>>>>>
>>>>>> A2 = rand(10, 15)
>>>>>> map(f, A2[1,:], A2[end,:]) # d == 1 -> 15 evaluations
>>>>>> map(f, A2[:,1], A2[:,end]) # d == 2 -> 10 evaluations
>>>>>>
>>>>>> A3 = rand(10, 15, 8)
>>>>>> map(f, A3[1,:,:], A3[end,:,:]) # d == 1 -> 15x8 evaluations
>>>>>> map(f, A3[:,1,:], A3[:,end,:]) # d == 2 -> 10x8 evaluations
>>>>>> map(f, A3[:,:,1], A3[:,:,end]) # d == 3 -> 10x15 evaluations
>>>>>>
>>>>>> I just want to consider one dimension at a time, so given A and d, 
>>>>>> and in this specific use case I don’t need to collect the results, so a 
>>>>>> for-loop without an allocated place for the answer instead of a map 
>>>>>> is just fine (probably preferrable, but it’s easier to go in that 
>>>>>> direction 
>>>>>> than in the other). What I’m struggling with, is how to generally 
>>>>>> formulate 
>>>>>> the indexing expressions (like [, 1, 
>>>>>> <size(A,d)-d instances of :>], but not in pseudo-code…). I assume 
>>>>>> this can be done somehow using CartesianIndexes and/or CartesianRanges, 
>>>>>> but I can’t get my mind around to how. Any help is much appreciated.
>>>>>>
>>>>>> // T
>>>>>> ​
>>>>>>
>>>>> ​
>>>
>>

[julia-users] Re: How do I, as efficiently as possible, iterate over the first and last elements along a given dimension of an array?

[Tomas Lycken]

…but not really. Reading the docstring more carefully:

Transform the given dimensions of array A using function f. f is called on 
each slice of A of the form A[…,:,…,:,…]. dims is an integer vector 
specifying where the colons go in this expression. The results are 
concatenated along the remaining dimensions. For example, if dims is [1,2] 
and A is 4-dimensional, f is called on A[:,:,i,j] for all i and j.

What I want to do, is rather call f on A[:,:,1,:] and A[:,:,end,:], but 
nothing in between 1 and end for that dimension. mapslices still eventually 
visit the entire array (either by slicing, or by iteration), but I only 
want to visit the “edges”. I might be missing something, though.

// T

On Thursday, March 24, 2016 at 1:48:36 AM UTC+1, Tomas Lycken wrote:

Yes, probably - thanks for the tip! I'll see if I can cook something up...
>
> On Thursday, March 24, 2016 at 1:45:32 AM UTC+1, Benjamin Deonovic wrote:
>>
>> Can mapslices help here?
>>
>>
>> On Wednesday, March 23, 2016 at 6:59:59 PM UTC-5, Tomas Lycken wrote:
>>>
>>> Is there an effective pattern to iterate over the “endpoints” of an 
>>> array along a given dimension?
>>>
>>> What I eventually want to accomplish is to apply a function (in this 
>>> case an equality test) to the two end points along a particular dimension 
>>> of an array. I think the pattern is easiest explained by considering 1D, 2D 
>>> and 3D:
>>>
>>> # assume the existence of some scalar-valued function f(x,y)
>>>
>>> A1 = rand(10)
>>> f(A1[1], A1[end]) # d == 1 (the only possible value) -> one evaluation
>>>
>>> A2 = rand(10, 15)
>>> map(f, A2[1,:], A2[end,:]) # d == 1 -> 15 evaluations
>>> map(f, A2[:,1], A2[:,end]) # d == 2 -> 10 evaluations
>>>
>>> A3 = rand(10, 15, 8)
>>> map(f, A3[1,:,:], A3[end,:,:]) # d == 1 -> 15x8 evaluations
>>> map(f, A3[:,1,:], A3[:,end,:]) # d == 2 -> 10x8 evaluations
>>> map(f, A3[:,:,1], A3[:,:,end]) # d == 3 -> 10x15 evaluations
>>>
>>> I just want to consider one dimension at a time, so given A and d, and 
>>> in this specific use case I don’t need to collect the results, so a 
>>> for-loop without an allocated place for the answer instead of a map is 
>>> just fine (probably preferrable, but it’s easier to go in that direction 
>>> than in the other). What I’m struggling with, is how to generally formulate 
>>> the indexing expressions (like [, 1, <size(A,d)-d 
>>> instances of :>], but not in pseudo-code…). I assume this can be done 
>>> somehow using CartesianIndexes and/or CartesianRanges, but I can’t get 
>>> my mind around to how. Any help is much appreciated.
>>>
>>> // T
>>> ​
>>>
>> ​

[julia-users] Re: How do I, as efficiently as possible, iterate over the first and last elements along a given dimension of an array?

[Tomas Lycken]

Yes, probably - thanks for the tip! I'll see if I can cook something up...

On Thursday, March 24, 2016 at 1:45:32 AM UTC+1, Benjamin Deonovic wrote:
>
> Can mapslices help here?
>
>
> On Wednesday, March 23, 2016 at 6:59:59 PM UTC-5, Tomas Lycken wrote:
>>
>> Is there an effective pattern to iterate over the “endpoints” of an array 
>> along a given dimension?
>>
>> What I eventually want to accomplish is to apply a function (in this case 
>> an equality test) to the two end points along a particular dimension of an 
>> array. I think the pattern is easiest explained by considering 1D, 2D and 
>> 3D:
>>
>> # assume the existence of some scalar-valued function f(x,y)
>>
>> A1 = rand(10)
>> f(A1[1], A1[end]) # d == 1 (the only possible value) -> one evaluation
>>
>> A2 = rand(10, 15)
>> map(f, A2[1,:], A2[end,:]) # d == 1 -> 15 evaluations
>> map(f, A2[:,1], A2[:,end]) # d == 2 -> 10 evaluations
>>
>> A3 = rand(10, 15, 8)
>> map(f, A3[1,:,:], A3[end,:,:]) # d == 1 -> 15x8 evaluations
>> map(f, A3[:,1,:], A3[:,end,:]) # d == 2 -> 10x8 evaluations
>> map(f, A3[:,:,1], A3[:,:,end]) # d == 3 -> 10x15 evaluations
>>
>> I just want to consider one dimension at a time, so given A and d, and 
>> in this specific use case I don’t need to collect the results, so a 
>> for-loop without an allocated place for the answer instead of a map is 
>> just fine (probably preferrable, but it’s easier to go in that direction 
>> than in the other). What I’m struggling with, is how to generally formulate 
>> the indexing expressions (like [, 1, <size(A,d)-d 
>> instances of :>], but not in pseudo-code…). I assume this can be done 
>> somehow using CartesianIndexes and/or CartesianRanges, but I can’t get 
>> my mind around to how. Any help is much appreciated.
>>
>> // T
>> ​
>>
>

[julia-users] How do I, as efficiently as possible, iterate over the first and last elements along a given dimension of an array?

[Tomas Lycken]

Is there an effective pattern to iterate over the “endpoints” of an array 
along a given dimension?

What I eventually want to accomplish is to apply a function (in this case 
an equality test) to the two end points along a particular dimension of an 
array. I think the pattern is easiest explained by considering 1D, 2D and 
3D:

# assume the existence of some scalar-valued function f(x,y)

A1 = rand(10)
f(A1[1], A1[end]) # d == 1 (the only possible value) -> one evaluation

A2 = rand(10, 15)
map(f, A2[1,:], A2[end,:]) # d == 1 -> 15 evaluations
map(f, A2[:,1], A2[:,end]) # d == 2 -> 10 evaluations

A3 = rand(10, 15, 8)
map(f, A3[1,:,:], A3[end,:,:]) # d == 1 -> 15x8 evaluations
map(f, A3[:,1,:], A3[:,end,:]) # d == 2 -> 10x8 evaluations
map(f, A3[:,:,1], A3[:,:,end]) # d == 3 -> 10x15 evaluations

I just want to consider one dimension at a time, so given A and d, and in 
this specific use case I don’t need to collect the results, so a for-loop 
without an allocated place for the answer instead of a map is just fine 
(probably preferrable, but it’s easier to go in that direction than in the 
other). What I’m struggling with, is how to generally formulate the 
indexing expressions (like [, 1, ], but not in pseudo-code…). I assume this can be done somehow using 
CartesianIndexes and/or CartesianRanges, but I can’t get my mind around to 
how. Any help is much appreciated.

// T
​

Re: [julia-users] Help needed regarding INERITANCE

[Tomas Lycken]

I've recently experimented with a macro that solves a related problem (although 
for different reasons). Basically it allows you to solve the same problems you 
could solve using polymorphism, but by easing the burden of composition. It 
might be helpful here too. 

https://github.com/tlycken/IJulia-Notebooks/blob/master/Using%20macros%20to%20implement%20value-objects%20and%20other%20simple%20wrappers%20in%20Julia.ipynb

// T 

[julia-users] Declaring a typealias that is also subtype of an abstract type

[Tomas Lycken]

Hah - only a couple of hours ago I uploaded this notebook, which outlines one 
possible way of getting there:

http://nbviewer.jupyter.org/github/tlycken/IJulia-Notebooks/blob/master/Using%20macros%20to%20implement%20value-objects%20and%20other%20simple%20wrappers%20in%20Julia.ipynb

// T 

Re: [julia-users] Re: Parametric splines?

[Tomas Lycken]

I tried googling for “parametric splines” but didn’t end up with a concise 
definition of what they are. If B-splines fit your needs (and it seems from 
the SciPy documentation that it might do), maybe Interpolations.jl 
 would be useful enough? The 
API is a little different, but I think this does what you’re after:

using Interpolations

t = 0:.1:.9
x = sin(2π*t)
y = cos(2π*t)
A = hcat(x,y)

itp = scale(interpolate(A, (BSpline(Cubic(Periodic())), NoInterp()), OnGrid()), 
t, 1:2)

tfine = 0:.01:1
xs, ys = [itp[t,1] for t in tfine], [itp[t,2] for t in tfine]

using Gadfly
plot(layer(x=x,y=y,Geom.point),layer(x=xs,y=ys,Geom.path))

Results on my machine:



// T

On Sunday, March 20, 2016 at 3:04:12 AM UTC+1, Kyle Barbary wrote:

Hi Kaj,
>
> A pull request adding a wrapper for this to Dierckx.jl would be most 
> welcome. This would be a matter of reading the docstring for the parcur 
> function here 
> 
>  
> and then writing a wrapper function that sets up the arguments correctly 
> and calls the Fortran function with ccall. There are a lot of examples in 
> Dierckx.jl. It’s a bit tedious but mostly straight forward. Fortunately 
> (for me anyway) knowing Fortran is not a requirement. I usually consult the 
> relevant scipy.interpolate wrapper (e.g., this one for splprep 
> )
>  
> to see how they handled things, and look at the tests in that package as 
> well.
>
> You can construct an array of vectors by prepending the element type, 
> which in your example would be Vector{Float64}. For example:
>
> julia> a = [1., 2.];
>
> julia> b = [3., 4.];
>
> julia> Vector{Float64}[a, b]
> 2-element Array{Array{Float64,1},1}:
>  [1.0,2.0]
>  [3.0,4.0]
>
> The plan for Julia 0.5, is that you won’t need to prepend the element 
> type: just [a, b] will do.
>
> By the way, collect is often not necessary. For example:
>
> julia> t = 0.0:0.1:0.5
> 0.0:0.1:0.5
>
> julia> sin(2*pi*t)
> 6-element Array{Float64,1}:
>  0.0
>  0.587785   
>  0.951057   
>  0.951057   
>  0.587785   
>  1.22465e-16
>
> Best,
> — Kyle
> ​
>
> On Sat, Mar 19, 2016 at 3:24 PM, Kaj Wiik  > wrote:
>
>> Replying to myself...sorry.
>>
>> It seems that the easiest way for now is to call SciPy:
>>
>> using PyCall
>> @pyimport scipy.interpolate as interpolate
>> t = collect(0:.1:1)
>> x = sin(2π*t)
>> y = cos(2π*t)
>> p = Array[]
>> push!(p, x)
>> push!(p, y)
>> tck, u = interpolate.splprep(p, s=0)
>> unew = collect(0:0.01:1)
>> out = interpolate.splev(unew, tck)
>>
>> using Winston
>> plot(x, y, "o", out[1], out[2], "-r")
>>
>> BTW, is there an easier way to create an array of vectors?
>>
>> Cheers,
>> Kaj
>>
>>
>> On Saturday, March 19, 2016 at 4:13:19 PM UTC+2, Kaj Wiik wrote:
>>>
>>>
>>> Is there a Julia package that implements parametric splines? 
>>>
>>> I noticed that the Dierckx Fortran library has an implementation but the 
>>> corresponding Julia package does not does not have bindings for it.
>>>
>>> Thanks,
>>> Kaj
>>>
>>
> ​

[julia-users] Re: what name would you give to the functional 'complement' of trunc()?

[Tomas Lycken]

I think stretch was a pretty good suggestion, actually.

trunc as in “truncate” makes sense for taking 3.14 -> 3 and -2.718 -> -2; 
you quite literally truncate the decimal representation of the number by 
chopping off the decimals. I could imagine the opposite, 3.14 -> 4 and 
-2.718 -> -3, as either “stretching” the numbers to reach all the way to 
the next integer, or “filling” up the remaining distance. Since fill is 
already loaded with quite different meaning, I’d go with stretch.

// T

On Saturday, March 12, 2016 at 2:04:23 PM UTC+1, Dan wrote:

trunc_to_inf and add an alias trunc_to_zero for regular trunc ??
>
> On Saturday, March 12, 2016 at 1:49:41 PM UTC+2, Jeffrey Sarnoff wrote:
>>
>> x = trunc(fp) yields fp rounded towards zero to the closest integer 
>> valued float.
>> What name would you give to the complementary function, rounding away 
>> from zero to the closest integer valued float?
>>
>> stretch(fp)  = signbit(fp) ? floor(fp) : ceil(fp)   # abs(trunc(fp) - 
>> stretch(fp)) is 0.0 when fp is integer valued and is 1.0 otherwise
>>
>>
>>
>> ​

[julia-users] Re: flatten arrays

[Tomas Lycken]

No, vec doesn't help because I have an array of arrays (actually a vector of 
vectors), not a multidimensional array. (I think vec would actually be a 
noop...) 

Flatten.jl might provide some useful idioms, but my application is json 
deserializing, so I can't restrict myself to fixed-size containers. I have to 
investigate my particular needs, though - it's possible that I don't need 
infinite-depth recursion at all, in which case it might be better to just do 
something hard-coded that solves the immediate problem. 

Thanks anyway! 

// T 

[julia-users] Re: good practice question: dispatch on type or on type instance

[Tomas Lycken]

I recently decided to switch the arguments in Interpolations.jl from types to 
instances, although in that case it actually made the interface a little uglier 
rather than prettier. The main reason was that I suddenly had the need for the 
flexibility provided by allowing for instance fields on the flags. My takeaway 
is that even if it's slightly more to type for the user, it's probably worth it 
in the long run. 

(there's also a note on this in the style guide section of the manual...) 

// T 

Re: [julia-users] flatten arrays

[Tomas Lycken]

This was the most recent mention of recursive array flattening that I could 
find - has anything happened since? Is this available somewhere (I couldn't 
find it in base)? 

// T 

Re: [julia-users] Meshgrid function

[Tomas Lycken]

The thing with meshgrid, is that it’s terribly inefficient compared to the 
equivalent idioms in Julia. Instead of implementing it and keeping with 
inefficient habits, embrace the fact that there are better ways to do 
things :)

Since this question is recurring, I think the problem might rather be 
documentation. As a user looking for a meshgrid function, where did you 
look? What did you look for, more specifically? Would it have helped to 
have some examples along the lines of “when you want to do this, using 
meshgrid, in MATLAB, here’s how to do the same thing without meshgrid in 
Julia”? I could turn the following writeup into a FAQ entry or something, 
but it would be nice to know where to place it to actually have it make a 
difference.
Making do without meshgrid (and getting out ahead!) 

For starters, if what you are after is a straight-up port, meshgrid is 
*very* easy to implement for yourself, so that you don’t have to think 
about the rest. *Mind you, this is not the most performant way to do this!*

meshgrid(xs, ys) = [xs[i] for i in 1:length(xs), j in length(ys)], [ys[j] for i 
in 1:length(xs), j in 1:length(ys)]

Try it out with X, Y = meshgrid(1:15, 1:8). (If you’re not happy with the 
orientation of the axes, just switch i and j in the meshgrid 
implementation.)

However, most of the uses of meshgrid in MATLAB is for stuff where you want 
to do a bunch of calculations for each point on a grid, and where each 
calculation is dependent on the coordinates of just one point. Usually 
something along the lines of

X, Y = meshgrid(xs, ys)
Z = sin(X) + 2 * Y.^2 ./ cos(X + Y)

In Julia, this is better done with a list comprehension altogether, 
eliminating the need for allocating X and Y:

Z = [sin(x) + 2 * y^2 / cos(x+y) for x in xs, y in ys]

If you think it’s now become too difficult to read what the actual 
transformation is, just refactor it out into a function:

foo(x,y) = sin(x) + 2 * y^2 / cos(x+y)
Z = [foo(x,y) for x in xs, y in ys]

If you do this a lot, you might want to avoid writing the list 
comprehension altogether; that’s easy too. Just create a function that does 
that for you. With a little clever interface design, you’ll see that this 
is even more useful than what you could do with meshgrid. 

inflate(f, xs, ys) = [f(x,y) for x in xs, y in ys]

# call version 1: with a named function
inflate(foo, xs, ys) 

# call version 2: anonymous function syntax
inflate((x,y) -> sin(x) + 2 * y^2 / cos(x,y), xs, ys)

# call version 3: with a do block
inflate(xs, ys) do x, y
sin(x) + 2 * y^2 / cos(x,y)
end

Note that the last version of this call is very flexible; in fact, it’s 
just as flexible as having a named function. For example, you can do 
everything in multiple steps, allowing for intermediate results that don’t 
have to allocate large arrays (as they would have, had you used meshgrid), 
making it easier to be explicit about whatever algorithm you’re using.

However, the two last versions of these calls are not the most performant 
as of yet, since they’re using anonymous functions. This is changing, 
though, and in 0.5 this won’t be an issue anymore. Until then, for large 
arrays I’d use the first call version, with a named function, to squeeze as 
much as possible out of this.

I did a small benchmark of this using a straight list comprehension (i.e. 
without the inflate function) vs using meshgrid. Here are the results:

julia> foo(x,y) = sin(x) + 2y.^2 ./ cos(x+y);

julia> function bench_meshgrid(N)
   X, Y = meshgrid(1:N, 1:N)
   foo(X, Y)
   end;

julia> function bench_comprehension(N)
   [foo(x,y) for x in 1:N, y in 1:N]
   end;

# warmup omitted

julia> @time bench_comprehension(1_000);
  0.033284 seconds (6 allocations: 7.630 MB)

julia> @time bench_meshgrid(1_000);
  0.071993 seconds (70 allocations: 68.666 MB, 5.12% gc time)

julia> @time bench_comprehension(10_000);
  3.547387 seconds (6 allocations: 762.940 MB, 0.06% gc time)

julia> @time bench_meshgrid(10_000);
ERROR: OutOfMemoryError()
 in .* at arraymath.jl:118
 in foo at none:1
 in bench_meshgrid at none:3

Note the memory usage: for the list comprehension, it’s entirely dominated 
by the result (10_000^2 * 8 / 1024^2 ~= 762.939 MB), while meshgrid eats 
almost ten times as much memory (the exact ratio will vary very much 
depending on the function foo and how many intermediate arrays it creates). 
For large inputs, meshgrid doesn't even make it.

// T

On Monday, March 7, 2016 at 5:35:13 PM UTC+1, Christoph Ortner wrote:

Have a look at 
>   http://matlabcompat.github.io
>
> If it doesn't have a mesh grid function, then maybe mesh grid from the 
> examples folder could be added to this package. 
>
> Christoph
>
​

Re: [julia-users] Re: [ANN] MathGL graphics library wrapper

[Tomas Lycken]

Just guessing by the name, but it seems to me that you might be able to 
leverage Interpolations.jl to get rid of the dependency on Splines.jl.

https://github.com/tlycken/Interpolations.jl 

// T 

Re: [julia-users] Re: Non-uniform interpolation in 1D

[Tomas Lycken]

Gridded here is the *interpolation scheme*, as opposed to (implicitly 
uniform) BSpline interpolation; it’s simply the way Interpolations.jl lets 
you specify which algorithm you want to use. Compare the following 
incantations:

itp = interpolate(A, BSpline(Linear()), OnGrid()) # linear b-spline 
interpolation; x is implicitly 1:length(A)
itp = interpolate((x,), A, Gridded(Linear())) # linear, "gridded" 
interpolation; x can be irregular
itp = interpolate(A, BSpline(Cubic(Flat())), OnCell()) # cubic b-spline with 
free boundary condition

That is; you give the data to be interpolated (A and, where applicable, x) 
as well as one or more arguments specifying the algortihm you want to use 
(for details on OnGrid/OnCell, see the readme). Gridded is what just we’ve 
called the family of algorithms that support irregular grids.

This is all documented in the readme, but documentation is not just about 
putting the information in writing - it’s also about putting it in the 
correct place, where it seems obvious to look for it. If you have 
suggestions on how this information can be made easier to find and digest, 
please file an issue or PR. All feedback is most welcome! :)

// T

On Sunday, February 28, 2016 at 7:37:42 PM UTC+1, Uwe Fechner wrote:

Hello,
>
> thanks for your explanations.
>
> Nevertheless I like the syntax of the package Dierckx much more.
> The expression Gridded(Linear()) is very confusing for me. What is gridded,
> in this example?
>
> Furthermore the Readme file of Interpolations is missing the information, 
> how
> to use it for the given problem.
>
> Best regards:
>
> Uwe
>
> On Saturday, February 27, 2016 at 4:57:07 PM UTC+1, Matt Bauman wrote:
>>
>> Interpolations is very similar, but it currently only supports linear and 
>> nearest-neighbor schemes for gridded interpolations:
>>
>> using Interpolations
>>
>> itp = interpolate((P_NOM,), ETA, Gridded(Linear())) # You pass the 
>> x-values as a tuple, since this generalizes to multi-dimensional coordinates
>> println(itp[3.5])
>>
>> x = linspace(1.5, 14.9, 1024)
>> y = itp[x]
>>
>> plot(x,y)
>>
>>
>>
>> On Saturday, February 27, 2016 at 10:10:28 AM UTC-5, Uwe Fechner wrote:
>>>
>>> Thanks. The following code works:
>>>
>>> using Dierckx
>>>
>>> P_NOM = [1.5, 2.2, 3.7, 5.6, 7.5, 11.2, 14.9]
>>> ETA   = [93., 94., 94., 95., 95., 95.5, 95.5]
>>> calc_eta = Spline1D(P_NOM, ETA, k=1)
>>>
>>> println(calc_eta(3.5))
>>>
>>> Nevertheless I would be interested how to do that with 
>>> Interpolations.jl. Sometimes you don't have Fortran available.
>>>
>>> Best regards:
>>>
>>> Uwe
>>>
>>> On Saturday, February 27, 2016 at 3:58:11 PM UTC+1, Yichao Yu wrote:



 On Sat, Feb 27, 2016 at 9:40 AM, Uwe Fechner  
 wrote:

> Hello,
>
> I don't think, that this works on a non-uniform grid. The array xg is 
> evenly spaced, and it 
> is NOT passed to the function InterpGrid.
>
>
 I've recently tried Dierckx which support non-uniform interpolation. I 
 only need very basic functions so I don't know if it has all the 
 flexibility you need but it's probably worth a look if you haven't.
  

> Uwe
>
>
> On Saturday, February 27, 2016 at 3:33:06 PM UTC+1, Cedric St-Jean 
> wrote:
>>
>> Hi Uwe,
>>
>> Have you tried Grid.jl? I haven't tried it, but this example looks 
>> like it might work with a non-uniform grid.
>>
>> # Let's define a quadratic function in one dimension, and evaluate it on 
>> an evenly-spaced grid of 5 points:
>> c = 2.3  # center
>> a = 8.1  # quadratic coefficient
>> o = 1.6  # vertical offset
>> qfunc = x -> a*(x-c).^2 + o
>> xg = Float64[1:5]
>> y = qfunc(xg)
>> yi = InterpGrid(y, BCnil, InterpQuadratic)
>>
>>
>>
>>
>> On Saturday, February 27, 2016 at 9:21:53 AM UTC-5, Uwe Fechner wrote:
>>>
>>> Hello,
>>>
>>> I am trying to port the following function from python to julia:
>>>
>>> # -*- coding: utf-8 -*-
>>> from scipy.interpolate import InterpolatedUnivariateSpline
>>> import numpy as np
>>> from pylab import plot
>>>
>>> P_NOM = [1.5, 2.2, 3.7, 5.6, 7.5, 11.2, 14.9]
>>> ETA   = [93., 94., 94., 95., 95., 95.5, 95.5]
>>>
>>> calc_eta = InterpolatedUnivariateSpline(P_NOM, ETA, k=1)
>>>
>>> # plotting code, only for testing
>>> if __name__ == "__main__":
>>> X = np.linspace(1.5, 14.9, 1024, endpoint=True)
>>> ETA = []
>>> for alpha in X:
>>> eta = calc_eta(alpha)
>>> ETA.append(eta)
>>> plot(X, ETA)
>>>
>>> The resulting plot is shown at the end of this posting.
>>>
>>> How can I port this to Julia?
>>>
>>> I am trying to use the package "Interpolations.jl", but I do not see 
>>> any
>>> example, that shows the interpolation on a non-uniform grid.
>>>
>>> For now I need only linear 

Re: [julia-users] Re: Non-uniform interpolation in 1D

[Tomas Lycken]

Higher than linear order interpolation on irregular grids is not supported in 
either Grid.jl or Interpolations.jl (and since the latter is a replacement for 
the former, it probably never will be supported in Grid). If you have a good 
scheme that you'd like to see implemented, do file an issue with some pointers 
to relevant resources that can be used as a baseline for an implementation.

Interpolations.jl now has a superset of the functionality in Grid.jl, but is (a 
lot!) faster, so if you're on Julia 0.4 (Interpolations.jl is not supported on 
0.3) there's no reason to use Grid.jl. 

// T 

Re: [julia-users] Are dataframes the best way to manipulate data?

[Tomas Lycken]

Re. something like C#’s LINQ, I think we will see a massive transformation 
of how people write Julia code once 0.5 arrives, since then we will have 
fast anonymous functions. This in turn will make functional-style idioms 
like collection pipelines 
 (an example of 
which are LINQ’s method syntax) much more common in various Julia code 
bases. For example, I think usage of the chaining operator |> will explode, 
compared to today.

When this happens, I think there will be numerous efforts to provide 
various utility methods that do similar things to e.g. C# LINQ extension 
methods. For example, someone will implement take(2) to return a function 
which, when applied in a method chain, returns the two first elements of a 
collection (perhaps even lazily). (This might even already exist, but the 
performance implications of using anonymous functions has so far hindered 
its widespread use.) Standard functions like map, collect etc will be 
extended to new data types, and new (or existing but little-used) functions 
will crop up to fill in the gaps in order to create this high-level API you 
are longing for. This will in turn make it easier for implementors of 
database adapters etc to hook into the emerging framework - and data 
consumers will have an API to work with that will (at least mostly) work 
regardless of whether the underlying data source is a DataFrame or a 
connection to an SQL database.

As you might be able to tell, I think the next Julia release is going to be 
really awesome…

// T

On Tuesday, February 23, 2016 at 2:42:10 PM UTC+1, Milan Bouchet-Valat 
wrote:

Le dimanche 21 février 2016 à 13:10 -0800, Shahbaz Chaudhary a écrit : 
> > I'm pretty new to Julia and only have marginally more experience with 
> > R so please excuse me if I don't understand something basic. 
> > 
> > According to Julia's website, the final api/syntax for manipulating 
> > data has not been finalized yet, although the momentum seems to be 
> > moving towards a dataframe style api. 
> I'm not sure what you base your impression on, but I'm not sure it's an 
> accurate description (see below). Anyway, two (o more) different APIs 
> can perfectly coexist. 
>
> > Since Julia is still a new language, doesn't it make sense to base 
> > the model on something closer to the relational algebra/sql/list 
> > comprehensions? I realize these three are not synonyms for each 
> > other, but relational algebra is supposed to have a more rigorous 
> > mathematical foundation in building the primitives used to manipulate 
> > data. SQL now has decades of use and has unarguably democratized data 
> > manipulation (I've seen lawyers and traders use sql, who would never 
> > use a full blown programming language).  
> > 
> > At least R's dataframe feel extremely clunky, although I'll admit 
> > that I may be missing something fundamental since Julia/Spark/Pandas 
> > seem to be adopting this model instead of the relational model. 
> > 
> > A language, built from the ground up to process datasets should have 
> > a more intuitive syntax. 
> > 
> > One potential issue is that relational algebra/sql don't handle 
> > ordered data well. I don't know enough about recent advances but 
> > surely an extension to relational primitives is more sound than 
> > adapting dataframes. 
> > 
> > Frankly I'm curious to learn from the more experienced here what I'm 
> > not understanding about dataframes and why they are so popular. 
> I think your feeling about a high-level API inspired by SQL is shared 
> by many of the developers involved with data frames and data analysis 
> in Julia. DataFrames can be one of the multiple backends which could be 
> manipulated using a common API similar to C#'s LINQ or to R's dplyr. 
> Things are still not settled, but you can have a look at this package: 
> https://github.com/JuliaStats/DataFramesMeta.jl 
>
> Also see this thread: 
> https://groups.google.com/d/msg/julia-dev/jL2FSL4EneE/c8vW8pdEqjYJ 
>
>
> Help is of course more than welcome in the area of database support! 
>
> Regards 
>
​

Re: [julia-users] How do I deprecate iteration for a custom iterator?

[Tomas Lycken]

Thanks, that gets me at least half the way!

However, I still can’t figure out how to give only one warning; if I give 
depwarn the symbols :start, :next and :done for the second argument, I get 
three warnings - if I give it something custom (and the same to all) I get 
three warnings *per element*. How do I give just one warning?

// T

On Sunday, February 21, 2016 at 4:53:19 PM UTC+1, Milan Bouchet-Valat wrote:

Le dimanche 21 février 2016 à 07:30 -0800, Tomas Lycken a écrit : 
> > In a package, I have a method foos which currently returns a 
> > collection of Bars. This is a little unfortunate, so I would like to 
> > instead return a value of type FooCollection on which I can call the 
> > bar function to get the array of Bars. I can get backwards 
> > compatibility by making FooCollection iterable, by implementing 
> > start, next and done. Now, I have the following: 
> > 
> > immutable Bar end 
> > immutable FooCollection 
> > bars::Vector{Bar} 
> > end 
> > 
> > foos() = FooCollection(Bar[]) 
> > bars(fc::FooCollection) = fc.bars 
> > 
> > start(fc::FooCollection) = start(fc.bars) 
> > next(fc::FooCollection, state) = next(fc.bars, state) 
> > done(fc::FooCollection, state) = done(fc.bars, state) 
> > This works - both the new and the old api:s are supported, and they 
> > do the right thing. 
> > 
> > Next, I’d like to deprecate iteration on FooCollection, to make it 
> > obvious to users that although their code isn’t broken, they’re 
> > relying on an API that might not work forever. Instead of for bar in 
> > foos() I want them to write for bar in bars(foos()). Thus, I make the 
> > following changes to the iteration protocol: 
> > 
> > @deprecate start(fc::FooCollection) start(bars(fc)) 
> > @deprecate next(fc::FooCollection, state) next(bars(fc), state) 
> > @deprecate done(fc::FooCollection, state) done(bars(fc), state) 
> > This works and gives deprecation messages when I write for bar in 
> > foos() println(bar) end, but the messages aren’t very helpful; for 
> > one thing, I get three of them (one for each method), and for another 
> > they refer to functions which aren’t present in the user’s code (and 
> > which the user won’t realize why they’re called unless they know how 
> > to implement iterators in Julia…). 
> > 
> > Is there a better/more helpful way to do this, without having to re- 
> > implement the @deprecate machinery entirely? 
> You can call depwarn() directly from inside these functions, and print 
> a custom message like "iteration over FooCollection is deprecated". See 
> base/deprecated.jl for examples. 
>
>
> Regards 
>
​

[julia-users] How do I deprecate iteration for a custom iterator?

[Tomas Lycken]

In a package, I have a method foos which currently returns a collection of 
Bars. This is a little unfortunate, so I would like to instead return a 
value of type FooCollection on which I can call the bar function to get the 
array of Bars. I can get backwards compatibility by making FooCollection 
iterable, by implementing start, next and done. Now, I have the following:

immutable Bar end
immutable FooCollection
bars::Vector{Bar}
end

foos() = FooCollection(Bar[])
bars(fc::FooCollection) = fc.bars

start(fc::FooCollection) = start(fc.bars)
next(fc::FooCollection, state) = next(fc.bars, state)
done(fc::FooCollection, state) = done(fc.bars, state)

This works - both the new and the old api:s are supported, and they do the 
right thing.

Next, I’d like to deprecate iteration on FooCollection, to make it obvious 
to users that although their code isn’t broken, they’re relying on an API 
that might not work forever. Instead of for bar in foos() I want them to 
write for bar in bars(foos()). Thus, I make the following changes to the 
iteration protocol:

@deprecate start(fc::FooCollection) start(bars(fc))
@deprecate next(fc::FooCollection, state) next(bars(fc), state)
@deprecate done(fc::FooCollection, state) done(bars(fc), state)

This works and gives deprecation messages when I write for bar in foos() 
println(bar) end, but the messages aren’t very helpful; for one thing, I 
get three of them (one for each method), and for another they refer to 
functions which aren’t present in the user’s code (and which the user won’t 
realize why they’re called unless they know how to implement iterators in 
Julia…).

Is there a better/more helpful way to do this, without having to 
re-implement the @deprecate machinery entirely?

// T
​

Re: [julia-users] Options for constructing a 3D surface from a point cloud

[Tomas Lycken]

There is a marching squares algorithm implemented in 
[Contour.jl](https://github.com/tlycken/Contour.jl) too; I have put some 
thoughts on API design in writing in [this 
issue](https://github.com/tlycken/Contour.jl/issues/29).

Meshing.jl might be a better place for these algorithm anyway (and the name 
"Contour.jl" is poorly chosen anyway - a plural form would have been 
better) so the question of moving the existing code into a different repo 
has already been raised independently). Maybe it's time to combine efforts? 
:)

I also saw discussions somewhere recently about a "marching triangles" 
algorithm for calculating contour curves on non-gridded data, which should 
probably live comfortably in the same space too.

// T

On Sunday, February 21, 2016 at 2:05:44 AM UTC+1, Steve Kelly wrote:
>
> Do you have a sample dataset? The algorithms for triangulating a signed 
> distance field can be found in Meshing.jl. I implemented Marching Cubes 
> recently and started Marching Squares today, but have yet to tag a release 
> because I need to settle on an API.
>
> I currently am working on solid modeling via implicit functions. More 
> generally I work in digital fabrication (Fab Lab) and would love to have 3D 
> scanning in the Julia workflow. If you can share more about the dataset you 
> have, I'll see if we can make it work with the tools we have available now. 
> On Feb 19, 2016 2:36 AM, "Igor"  wrote:
>
>> Chris , I'm interested in this area too.  Please post here if you come up 
>> with some solution you would like.
>>
>> Best regards, Igor
>
>

Re: [julia-users] Appveyor: Private repos using private repos

[Tomas Lycken]

I'm just guessing here, but I think it could work if you specify the url 
using https rather than the git protocol. That is, 
https://github.com/EricForgy/Pages.jl.git instead of 
g...@github.com:EricForgy/Pages.jl.git.

That error message could be made a lot better though, by just listing the 
supported protocols. (That might not be as easy as it sounds, though; after 
skimming the source, it seems to me that this error message must come from 
inside libgit2...)

// T

On Saturday, February 20, 2016 at 6:27:53 PM UTC+1, Eric Forgy wrote:
>
> Thanks Isaiah.
>
> I bit the bullet and tried the "ssh stuff". I was able to get it to work 
> and my tests passed for Julia 0.4 (yay!), but the tests on latest master 
> failed with an error:
>
> INFO: Cloning Pages from g...@github.com:EricForgy/Pages.jl.git
> ERROR: GitError(Code:ERROR, Class:Net, Unsupported URL protocol)
>
> Was I just unlucky or is there a difference between 0.4 and 0.5 with 
> regard to URL formats?
>
> In 0.4, this worked: 
>
> Pkg.clone("g...@github.com:EricForgy/Pages.jl.git")
>
> But, from the error message, it seems this isn't working on 0.5?
>
> Any idea?
>
> On Saturday, February 20, 2016 at 10:19:00 PM UTC+8, Isaiah wrote:
>>
>> That error means git is trying to show a username prompt, but can't 
>> because there is no input device attached. You need the "ssh stuff" so that 
>> git on the appveyor build machine can be authenticated to clone the 
>> second private repo.
>>
>

[julia-users] Re: Convert a collection to an array

[Tomas Lycken]

Sorry, I misread your dimensions. vcat(ySol...) will solve the problem 
correctly for you :)

julia> ySol = [rand(2)' for _ in 1:20]
20-element Array{Array{Float64,2},1}:
 1x2 Array{Float64,2}:
 0.751346  0.212002  
...
 1x2 Array{Float64,2}:
 0.52188  0.121669   

julia> vcat(ySol...)
20x2 Array{Float64,2}:
 0.7513460.212002 
...
 0.52188 0.121669

// T

On Thursday, February 18, 2016 at 7:35:15 PM UTC+1, Tomas Lycken wrote:

If it’s not longer than that, you can easily do that with hcat:
>
> julia> ySol = [rand(2) for _ in 1:20]
> 20-element Array{Array{Float64,1},1}:
>  [0.6383963589240325,0.952443724507759]  
>  [0.8543445393734637,0.5497614848396764] 
>  [0.7180883594522589,0.2699980988282249] 
>  [0.879654133666188,0.008079464648794499]
>  [0.32422181761199975,0.2699464303552557]
>  [0.26787882268244223,0.4271951131306577]
>  [0.2613014121460324,0.9629053122741955] 
>  [0.3473409046077083,0.8228842149724882] 
>  [0.9074984806753585,0.5398222579191083] 
>  [0.850898493473214,0.10093766734240517] 
>  [0.7103469388877257,0.6295378284694306] 
>  [0.7598286632595868,0.9313542937160446] 
>  [0.7174343129435241,0.9971075729799896] 
>  [0.49861954369803874,0.3683763032284182]
>  [0.8145463614876596,0.616519930503598]  
>  [0.6922945697041882,0.646282075186712]  
>  [0.8108510573789565,0.20586283678616368]
>  [0.46599068004928035,0.7351678286521091]
>  [0.6932932362204158,0.8608833557510405] 
>  [0.1747785273075091,0.9016114746811359] 
>
> julia> hcat(ySol...)'
> 20x2 Array{Float64,2}:
>  0.638396  0.952444  
>  0.854345  0.549761  
>  0.718088  0.269998  
>  0.879654  0.00807946
>  0.324222  0.269946  
>  0.267879  0.427195  
>  0.261301  0.962905  
>  0.347341  0.822884  
>  0.907498  0.539822  
>  0.850898  0.100938  
>  0.710347  0.629538  
>  0.759829  0.931354  
>  0.717434  0.997108  
>  0.49862   0.368376  
>  0.814546  0.61652   
>  0.692295  0.646282  
>  0.810851  0.205863  
>  0.465991  0.735168  
>  0.693293  0.860883  
>  0.174779  0.901611
>
> Note, however, that splatting like this isn’t very performant (yet…) so if 
> you need to do this in a tight loop, or for a much longer array ySol, you 
> should consider other options.
>
> // T
>
> On Thursday, February 18, 2016 at 7:23:55 PM UTC+1, Chris Nabold wrote:
>
>
>>
>> I have a collection ySol:
>> julia> ySol=collect(ySol)
>> 21-element Array{Any,1}:
>>  1x2 Array{Float64,2}:
>>  9000.0  0.0
>>  1x2 Array{Float64,2}:
>>  8998.79  -4.80427
>>  1x2 Array{Float64,2}:
>>  8995.29  -9.0677
>>  1x2 Array{Float64,2}:
>>  8989.86  -12.4678
>>  1x2 Array{Float64,2}:
>>  8982.97  -14.9547
>>  1x2 Array{Float64,2}:
>>  8975.04  -16.66
>>  1x2 Array{Float64,2}:
>>  8966.41  -17.7772
>>  1x2 Array{Float64,2}:
>>  8957.33  -18.4864
>>  1x2 Array{Float64,2}:
>>  8947.97  -18.9266
>>  1x2 Array{Float64,2}:
>>  8938.43  -19.195
>>  1x2 Array{Float64,2}:
>>  8928.79  -19.3559
>>  1x2 Array{Float64,2}:
>>  8919.09  -19.4502
>>  1x2 Array{Float64,2}:
>>  8909.35  -19.5037
>>  1x2 Array{Float64,2}:
>>  8899.59  -19.5323
>>  1x2 Array{Float64,2}:
>>  8889.82  -19.5457
>>  1x2 Array{Float64,2}:
>>  8880.04  -19.5498
>>  1x2 Array{Float64,2}:
>>  8870.27  -19.5484
>>  1x2 Array{Float64,2}:
>>  8860.5  -19.5435
>>  1x2 Array{Float64,2}:
>>  8850.73  -19.5366
>>  1x2 Array{Float64,2}:
>>  8840.96  -19.5285
>>  1x2 Array{Float64,2}:
>>  8831.2  -19.5196
>> How can I convert it into an array of dimension 20,2
>> eg yy=zeros(20,2)
>> yy=ySol
>> Thank you for your help
>>
> ​
>
​

[julia-users] Re: Convert a collection to an array

[Tomas Lycken]

If it’s not longer than that, you can easily do that with hcat:

julia> ySol = [rand(2) for _ in 1:20]
20-element Array{Array{Float64,1},1}:
 [0.6383963589240325,0.952443724507759]  
 [0.8543445393734637,0.5497614848396764] 
 [0.7180883594522589,0.2699980988282249] 
 [0.879654133666188,0.008079464648794499]
 [0.32422181761199975,0.2699464303552557]
 [0.26787882268244223,0.4271951131306577]
 [0.2613014121460324,0.9629053122741955] 
 [0.3473409046077083,0.8228842149724882] 
 [0.9074984806753585,0.5398222579191083] 
 [0.850898493473214,0.10093766734240517] 
 [0.7103469388877257,0.6295378284694306] 
 [0.7598286632595868,0.9313542937160446] 
 [0.7174343129435241,0.9971075729799896] 
 [0.49861954369803874,0.3683763032284182]
 [0.8145463614876596,0.616519930503598]  
 [0.6922945697041882,0.646282075186712]  
 [0.8108510573789565,0.20586283678616368]
 [0.46599068004928035,0.7351678286521091]
 [0.6932932362204158,0.8608833557510405] 
 [0.1747785273075091,0.9016114746811359] 

julia> hcat(ySol...)'
20x2 Array{Float64,2}:
 0.638396  0.952444  
 0.854345  0.549761  
 0.718088  0.269998  
 0.879654  0.00807946
 0.324222  0.269946  
 0.267879  0.427195  
 0.261301  0.962905  
 0.347341  0.822884  
 0.907498  0.539822  
 0.850898  0.100938  
 0.710347  0.629538  
 0.759829  0.931354  
 0.717434  0.997108  
 0.49862   0.368376  
 0.814546  0.61652   
 0.692295  0.646282  
 0.810851  0.205863  
 0.465991  0.735168  
 0.693293  0.860883  
 0.174779  0.901611

Note, however, that splatting like this isn’t very performant (yet…) so if 
you need to do this in a tight loop, or for a much longer array ySol, you 
should consider other options.

// T

On Thursday, February 18, 2016 at 7:23:55 PM UTC+1, Chris Nabold wrote:


>
> I have a collection ySol:
> julia> ySol=collect(ySol)
> 21-element Array{Any,1}:
>  1x2 Array{Float64,2}:
>  9000.0  0.0
>  1x2 Array{Float64,2}:
>  8998.79  -4.80427
>  1x2 Array{Float64,2}:
>  8995.29  -9.0677
>  1x2 Array{Float64,2}:
>  8989.86  -12.4678
>  1x2 Array{Float64,2}:
>  8982.97  -14.9547
>  1x2 Array{Float64,2}:
>  8975.04  -16.66
>  1x2 Array{Float64,2}:
>  8966.41  -17.7772
>  1x2 Array{Float64,2}:
>  8957.33  -18.4864
>  1x2 Array{Float64,2}:
>  8947.97  -18.9266
>  1x2 Array{Float64,2}:
>  8938.43  -19.195
>  1x2 Array{Float64,2}:
>  8928.79  -19.3559
>  1x2 Array{Float64,2}:
>  8919.09  -19.4502
>  1x2 Array{Float64,2}:
>  8909.35  -19.5037
>  1x2 Array{Float64,2}:
>  8899.59  -19.5323
>  1x2 Array{Float64,2}:
>  8889.82  -19.5457
>  1x2 Array{Float64,2}:
>  8880.04  -19.5498
>  1x2 Array{Float64,2}:
>  8870.27  -19.5484
>  1x2 Array{Float64,2}:
>  8860.5  -19.5435
>  1x2 Array{Float64,2}:
>  8850.73  -19.5366
>  1x2 Array{Float64,2}:
>  8840.96  -19.5285
>  1x2 Array{Float64,2}:
>  8831.2  -19.5196
> How can I convert it into an array of dimension 20,2
> eg yy=zeros(20,2)
> yy=ySol
> Thank you for your help
>
​

[julia-users] Re: Coordinates of graph layout algorithms

[Tomas Lycken]

Without having looked at the code, I'd say that such a separation effort is 
definitely the way to go (in fact, I created the Contours.jl package for very 
similar reasons, based on code originally in Gadfly). 

I would even think it makes sense to move the Compose-based drawing into 
Gadfly, to make the GraphLayouts package independent of drawing backend (and 
thus reusable by other plotting packages too). This might be a too big 
operation to handle in one go, though. 

// T 

Re: [julia-users] Re: how to undefine variable in composite types

[Tomas Lycken]

My main concern is predictability. 

Imagine that I define a type Foo and start playing around with it - define some 
functions that work on foot, perhaps one or two types that hold foos, and 
manipulate a bunch of Foo instances.

Then, I redefine the Foo type - say, I add a field. Which of the other things 
that I have defined are still valid?

If I understand the motivation for this feature correctly, it's to make 
exploratory programming easier. However, I think it would actually make it a 
lot more difficult - it would basically give you a bazooka so heavy that the 
only thing you can reasonably aim at is your own foot... 

//T 

[julia-users] Re: Julia vs Matlab: interpolation and looping

[Tomas Lycken]

I'd love to add non-uniform interpolation schemes of higher degree than linear 
to Interpolations.jl - the two main reasons for why it hasn't been done already 
are time (focus has been on reaching feature parity with Grid.jl, for which the 
package started out as a replacement effort) and knowledge (I don't know about 
good algorithms for it). The second one is easily solved if you can link to a 
few good papers on the subject and/or liberally licensed implementations - the 
first one is best fixed with a PR :) 

// T 

[julia-users] write julia consolle output in a text file

[Tomas Lycken]

For example 

open("log.txt", "w") do f
  write(f," Hello, text file!")
end 

or 

julia your-script.jl > log.txt

depending on how you invoke your Julia code and what you mean by "console 
output". 

// T 

[julia-users] Using Interpolations.jl How to define the behavior when out of bounds?

[Tomas Lycken]

Yes, that's the correct way to specify the extrapolation behavior, but you 
don't seem to use it afterwards; your last two lines refer to Uinterp and 
Vinterp, rather than Uextrap and Vextrap. To get the desired behavior for OOB 
indices, you must index into the result of extrapolate(...). 

If you want to avoid this, you can of course do both interpolation and 
extrapolation in one step:

itp = extrapolate(interpolate(...), NaN)
itp[-3.5] # NaN

// T 

[julia-users] Great read as Julia's package ecosystem grows ("a la npm" nested dependencies)

[Tomas Lycken]

Interesting read (or skim, in my case)! 

I think there are some major problems with npm's nested approach too - for 
example, file paths tend to become so long that at least Windows has real 
troubles handling them (it's not fun to try to get out of a situation where you 
can't delete a folder because it's tree of subfolders is too deep, and not the 
subfolders because their paths are too long). 

My main takeaway, though, is that our package manager lacks solutions to two 
problems that npm solves with nested dependencies inside the project folder:

1 The ability to have different versions of the same package active on the same 
machine, and even in the same project's (sub)dependencies. 

2. The ability to specify dependencies and their versions on a per-project 
basis, also for projects that are not modules themselves, allowing you to check 
in a file specifying dependencies and then running something like Pkg.restore() 
to install everything you need. 

I think this can definitely be done with smaller changes to our ecosystem than 
going npm-style all the way, but I also think that these are important problems 
that should be considered in a future revamp of our package manager. 

// T 

Re: [julia-users] What's the "correct" way to handle optional bounds-checking?

[Tomas Lycken]

Interesting read, thanks!

I guess I'll hold off implementing this until 0.5 lands, then.

// T

On Monday, January 25, 2016 at 7:56:15 PM UTC+1, Yichao Yu wrote:
>
> On Mon, Jan 25, 2016 at 1:49 PM, Tomas Lycken <tomas@gmail.com 
> > wrote: 
> > On e.g. Arrays, I can index with @inbounds to avoid bounds checking. In 
> my 
> > own custom type, which also implements getindex, what is the correct way 
> of 
> > leveraging inbounds? 
> > 
> > For example, I have code now that looks something like this: 
> > 
> > function getindex(A::MyCustomArray, x) 
> > ix = clamp(round(Int, x), 1, length(A)) # yes, clamp makes sense in 
> this 
> > case 
> > ... 
> > 
> > but if the caller has specified @inbounds, I want to avoid the clamp and 
> > just set ix = round(Int, x). What would be the correct way to express 
> that? 
>
> No automatic way to do this on 0.4 AFAIK. For 0.5-dev see 
> https://github.com/JuliaLang/julia/pull/14474 
>
> > 
> > // T 
>

[julia-users] What's the "correct" way to handle optional bounds-checking?

[Tomas Lycken]

On e.g. Arrays, I can index with @inbounds to avoid bounds checking. In my 
own custom type, which also implements getindex, what is the correct way of 
leveraging inbounds?

For example, I have code now that looks something like this:

function getindex(A::MyCustomArray, x)
ix = clamp(round(Int, x), 1, length(A)) # yes, clamp makes sense in this 
case
...

but if the caller has specified @inbounds, I want to avoid the clamp and 
just set ix = round(Int, x). What would be the correct way to express that?

// T
​

[julia-users] Re: problems reading a file

[Tomas Lycken]

Try converting from binary to text with UTF8String instead of ASCIIString - 
then all characters should display correctly. 

//T 

Re: [julia-users] Re: Code review request: interacting with another program via stdin, stdout, stderr.

[Tomas Lycken]

The other global variables actually hold data, which changes according to 
the values read from the different streams. Inspired by the wrapper idea, I 
wrapped them in a type so that it becomes clear that they are always 
strings; hopefully this will help with performance.

You won’t see any performance difference from just wrapping them. However, 
since you can now rebind the fields of the wrappers (which have strictly 
specified types) instead of rebinding the variables themselves, all those 
variables can now be marked const. This should give you some additional 
speed gain.

Another thing to consider, is that you probably want to be able to do this 
with several commands in parallel. (Consider, for example, running multiple 
echo programs and passing stuff between them) For that, you would need to 
wrap all of this in a function anyway. I would consider the following API:

function popen3(f::Function, cmd::Cmd)
# setup all streams, like your current popen3 function does

# also, create all state handling for the different streams

# finally, pass suitable arguments to f to let the user provide the actual 
action
f(pin.in, out.out, err.out) # for example, maybe something else makes more 
sense
end

The caller can then use a do block to consume this:

popen3(`./inout`) do pin, pout, perr
# use the streams here
end

Since it’s all now wrapped in a function, “globals” won’t be a performance 
problem anymore. As a bonus, you can work with several, parallel, 
invocations without conflict between them. You might still see some 
slowness due to passing anonymous functions, but it’s a) probably 
negligible compared to the IO cost, and b) it’s actively being worked on 
<https://github.com/JuliaLang/julia/pull/13412> and will eventually be fast.

// T

On Tuesday, January 12, 2016 at 3:50:04 AM UTC+1, Miguel Bazdresch wrote:

Tomas,
>
> Thanks for your feedback! I couldn't find a package that handles this 
> already. If this technique proves to work reliably in Gaston, I'll be happy 
> to put it in a package.
>
> Regarding `readnow`, I never rebind it. It doesn't hold any actual value; 
> it is just a `Condition()`, which can be used to notify asynchronous tasks. 
> I followed your advice and made it a `const`.
>
> The other global variables actually hold data, which changes according to 
> the values read from the different streams. Inspired by the wrapper idea, I 
> wrapped them in a type so that it becomes clear that they are always 
> strings; hopefully this will help with performance.
>
> The latest version of the code is at 
> https://gist.github.com/mbaz/bb7e2cbaaecc031b1d88
>
> -- mb
>
> On Mon, Jan 11, 2016 at 9:05 AM, Tomas Lycken <tomas@gmail.com 
> > wrote:
>
>> Interesting question! If you find a good approach to do this, wrapping it 
>> in a package is certainly interesting (have you checked that there aren’t 
>> any packages that handle this already?).
>>
>> I can’t help much with the threading stuff, but regarding your global 
>> variable it should be possible to work around the slowness of that. I don’t 
>> know how, or when, you re-bind readnow, but there are two ways to fix it 
>> depending on the specifics:
>>
>>1. 
>>
>>You never rebind readnow, just mutate it. Mark it const, and it’s 
>>fast. (const means *impossible to rebind*, not *impossible to mutate*)
>>2. 
>>
>>You rebind readnow, and Condition() is mutable: Mark it const and 
>>mutate it instead of rebinding.
>>3. 
>>
>>You rebind readnow and Condition() is immutable: Wrap the Condition 
>>in a mutable type, which you assign to const readnow instead, and 
>>then rebind the field in this mutable type. Something like this:
>>
>> type ConditionWrapper
>> c::Condition
>> end
>>
>> const readnow = ConditionWrapper(Condition())
>>
>> # where you update:
>> readnow.c = Condition(args...)
>>
>> This all assumes that Condition is an immutable concrete type, and you 
>> just want to switch it out for an instance with other field values. If you 
>> actually need to change the *type* of readnow, all bets are off and this 
>> trick won’t work.
>>
>> // T
>>
>> On Friday, January 8, 2016 at 9:28:02 PM UTC+1, Miguel Bazdresch wrote:
>>
>> Hello,
>>>
>>> I'd be grateful if you could take a look at some code and suggest 
>>> improvements.
>>>
>>> I'm trying to interact with a long-lived process (gnuplot). This process 
>>> reads commands from its STDIN; after each command is executed, it produces 
>>> output on either its STDOUT or STDERR. It's impossible to predict ahead of

[julia-users] Re: Code review request: interacting with another program via stdin, stdout, stderr.

[Tomas Lycken]

Interesting question! If you find a good approach to do this, wrapping it 
in a package is certainly interesting (have you checked that there aren’t 
any packages that handle this already?).

I can’t help much with the threading stuff, but regarding your global 
variable it should be possible to work around the slowness of that. I don’t 
know how, or when, you re-bind readnow, but there are two ways to fix it 
depending on the specifics:

   1. 
   
   You never rebind readnow, just mutate it. Mark it const, and it’s fast. (
   const means *impossible to rebind*, not *impossible to mutate*)
   2. 
   
   You rebind readnow, and Condition() is mutable: Mark it const and mutate 
   it instead of rebinding.
   3. 
   
   You rebind readnow and Condition() is immutable: Wrap the Condition in a 
   mutable type, which you assign to const readnow instead, and then rebind 
   the field in this mutable type. Something like this:
   
type ConditionWrapper
c::Condition
end

const readnow = ConditionWrapper(Condition())

# where you update:
readnow.c = Condition(args...)

This all assumes that Condition is an immutable concrete type, and you just 
want to switch it out for an instance with other field values. If you 
actually need to change the *type* of readnow, all bets are off and this 
trick won’t work.

// T

On Friday, January 8, 2016 at 9:28:02 PM UTC+1, Miguel Bazdresch wrote:

Hello,
>
> I'd be grateful if you could take a look at some code and suggest 
> improvements.
>
> I'm trying to interact with a long-lived process (gnuplot). This process 
> reads commands from its STDIN; after each command is executed, it produces 
> output on either its STDOUT or STDERR. It's impossible to predict ahead of 
> time which of the two streams will be used.
>
> To simplify my tests, I wrote an "echo server" in C, which reads a 
> character from its STDIN and outputs it again over either STDOUT or STDERR. 
> The code is here: https://gist.github.com/mbaz/1e242694a9c4f1eca576
>
> Then I wrote a julia program that reads a character from its own STDIN, 
> sends it to the C echo server, and then tries to read the server's STDOUT 
> and STDERR until it finds a character. The code is here: 
> https://gist.github.com/mbaz/bb7e2cbaaecc031b1d88
>
> This code works, but I don't know if it is the best approach. Two specific 
> questions I have:
>
> * Is my `popen3()` function necessary? This closed issue seems to suggest 
> it's not, but I can't figure out how to accomplish what I need in a more 
> simple manner: https://github.com/JuliaLang/julia/issues/11824
>
> * Are global variables required to share data with asynchronous tasks? 
> Since global variables are slow, this approach may produce undesired code 
> slowdowns.
>
> Thanks,
>
> -- mb
>
​

[julia-users] PriorityQueue allowing multiple elements with same priority

[Tomas Lycken]

I tried this today:

immutable State
a::Int
b::Int
end

import Base.Collections: PriorityQueue, enqueue!, dequeue!, peek

pq = PriorityQueue(Int,Int)

enqueue!(pq, State(5,3), 4)
enqueue!(pq, State(5,3), 2)

but it fails with the error

LoadError: ArgumentError: PriorityQueue keys must be unique

I assume this is because the states are immutable, and thus compare equal 
if all their values compare equal. Is there a way to make a priority queue 
insert the same state on several priorities, i.e. to support my example 
above?

// T
​

[julia-users] Re: Iterating over leafs of a tree: use Cartesian?

[Tomas Lycken]

I don’t think so; your way of determining the loop ranges is a little bit 
too specific.

The only two ways of generating loop bounds is either from a variable 
referencing an array (or array-like object), in which case the bounds will 
be determined by the size of the array, or with an anonymous function. The 
anonymous function allows for a limited set of rewrite-rules based on the 
dimension (which is its input), but it doesn’t allow referencing another 
dimension AFAICS. The closest I could get was this, which references the 
wrong dimension:

julia> macroexpand(quote
@nloops 3 i d->0:n-i_d begin
sum(@ntuple 3 i)
end
end)

quote  # In[15], line 2:
begin  # cartesian.jl, line 31:
for i_3 = 0:n - i_3 # cartesian.jl, line 32:
nothing # cartesian.jl, line 33:
begin  # cartesian.jl, line 31:
for i_2 = 0:n - i_2 # cartesian.jl, line 32:
nothing # cartesian.jl, line 33:
begin  # cartesian.jl, line 31:
for i_1 = 0:n - i_1 # cartesian.jl, line 32:
nothing # cartesian.jl, line 33:
begin  # In[15], line 3:
sum((i_1,i_2,i_3))
end # cartesian.jl, line 34:
nothing
end
end # cartesian.jl, line 34:
nothing
end
end # cartesian.jl, line 34:
nothing
end
end
end

I guess you’ve already found it, but the full docs of Base.Cartesian are 
enlightening, especially the reference at the bottom: 
http://docs.julialang.org/en/release-0.4/devdocs/cartesian/#devdoc-cartesian-reference

// T

On Thursday, January 7, 2016 at 11:10:18 AM UTC+1, Asbjørn Nilsen Riseth 
wrote:

I would like to loop over all the \beta_i values of the following tree. 
>
>
> 
>
> What is the best Julia way of doing this? I'm also interested in 
> generalising this to N levels.
>
>
> For the given tree, the following works. *Is there a way to use @nloops 
> to shorten this?*
> n = Int(1/0.2)
> for i_3 = 0:n
> for i_2 = 0:n-i_3
> for i_1 = 0:n-i_2-i_3
> i_0 = n-sum(@ntuple 3 i)
> β = [i_3, i_2, i_1, i_0]/n
> @show β
> end
> end
> end
>
>
> ​

Re: [julia-users] Re: what's the difference btwn multiple dispatch and c++ overloading?

[Tomas Lycken]

There's also the complication that "run time" and "compile time" are much 
easier to define in C++ than in Julia...

// T

On Wednesday, December 23, 2015 at 2:21:07 PM UTC+1, Stefan Karpinski wrote:
>
> That's correct. Semantically, multiple dispatch is resolved at run time, 
> but in practice, Julia is able to resolve a lot of calls completely 
> statically. Even more so than C++ because instead of generating a single 
> method body for an abstract type and using virtual dispatch to call methods 
> on instances of the abstract type, Julia aggressively specializes methods 
> on specific argument types, which eliminates the virtual dispatch inside 
> altogether. This is actually a fairly standard optimization in modern C++ 
> compilers as well, commonly known as devirtualization. So C++ compilers 
> will do this sometimes, but Julia takes this approach by default.
>
> On Wed, Dec 23, 2015 at 8:04 AM, Christof Stocker  > wrote:
>
>> To follow up with a question on this "multiple dispatch is resolved at 
>> runtime" statement:
>>
>> I was under the impression, that if the Julia JIT compiler is able to 
>> infer the concrete types of the variables involved in a method call, that 
>> it will select the appropriate method at compile time. Am I correct in this 
>> assumption, or did I misunderstand something?
>>
>>
>> On 2015-12-23 13:56, Eric Forgy wrote:
>>
>> Try this? 
>>
>>
>> http://stackoverflow.com/questions/1801216/what-is-the-difference-between-multiple-dispatch-and-method-overloading
>>
>> On Wednesday, December 23, 2015 at 8:50:08 PM UTC+8, Neal Becker wrote: 
>>>
>>> Is there a difference? 
>>>
>>>
>>
>

Re: [julia-users] How to change the character at a specific position of a string?

[Tomas Lycken]

If/when you need the result, you can use join to get a string. For example:

julia> as = collect("")
4-element Array{Char,1}:
 'a'
 'a'
 'a'
 'a'

julia> as[2] = 'b'
'b'

julia> as
4-element Array{Char,1}:
 'a'
 'b'
 'a'
 'a'

julia> join(as)
"abaa"

// T

On Tuesday, December 22, 2015 at 11:20:46 PM UTC+1, Erik Schnetter wrote:

You can use an array of characters instead of a string. This allows you to 
> modify individual characters as necessary. Alternatively, an array of UInt8 
> make make sense (if you only need ASCII characters).
>
> -erik
>
> On Tue, Dec 22, 2015 at 4:43 PM, Stefan Karpinski  > wrote:
>
>> Strings are immutable. There is no API to do this. You can construct a 
>> new string object by doing s[1:1]*"b"*s[3:end].
>>
>> On Tue, Dec 22, 2015 at 4:06 PM, Dongning Guo > > wrote:
>>
>>> Simple question: How to change the character at a specific position of a 
>>> string?  Searched for a while in vain.  Thanks.
>>>
>>>
>>> For example:
>>>
>>> *julia> **s=^("a",4)*
>>>
>>> *""*
>>> How to change the second character to *'**b'*, that is to change s to:
>>>
>>> *"abaa"*
>>>
>>>
>>>
>>>
>>>
>>
>
>
> -- 
> Erik Schnetter  
> http://www.perimeterinstitute.ca/personal/eschnetter/
>
​

[julia-users] Re: Array view without a single element

[Tomas Lycken]

For completeness, the solution I ended up using was to pop! a coin from the 
list, then call the method recursively *twice* - once for the case when the 
coin is used (and the recursive total is thus reduced by the coin’s value) 
and one where it’s not (and the total stays the same). Before returning, I 
push! the item back on the list.

// T

On Monday, December 21, 2015 at 9:33:17 PM UTC+1, Tomas Lycken wrote:

The algorithm I want to implement is fairly simple - it's a basic change 
> making problem: "given the following coins [20 coins with integer-values 
> between 1 and 50] how many ways is there to combine them to get a total 
> value of 150? two coins with equal value contribute one time each to a 
> combination involving only one of them."
>
> My approach was based on this SO answer 
> <http://stackoverflow.com/a/14993011/38055>, basically figuring it out 
> based on how many ways there are to get a total value of 150-x from the 
> list of all coins except x. I might be able to remove and re-insert the 
> value, but ordering is important (since I'll loop over the values, and I 
> only want to test each value once on each level).
>
> I'll have to think about this some more, but at least now I know that my 
> naïve approach is impossible to optimize regardless of my inability to 
> construct the indexing array efficiently :)
>
> Thanks!
>
> // T
>
> On Monday, December 21, 2015 at 6:57:39 PM UTC+1, Tero Frondelius wrote:
>>
>> Hi
>> Can you use a macro to produce your twenty nested loops? 
>
> ​

[julia-users] Re: Array view without a single element

[Tomas Lycken]

The algorithm I want to implement is fairly simple - it's a basic change 
making problem: "given the following coins [20 coins with integer-values 
between 1 and 50] how many ways is there to combine them to get a total 
value of 150? two coins with equal value contribute one time each to a 
combination involving only one of them."

My approach was based on this SO answer 
, basically figuring it out 
based on how many ways there are to get a total value of 150-x from the 
list of all coins except x. I might be able to remove and re-insert the 
value, but ordering is important (since I'll loop over the values, and I 
only want to test each value once on each level).

I'll have to think about this some more, but at least now I know that my 
naïve approach is impossible to optimize regardless of my inability to 
construct the indexing array efficiently :)

Thanks!

// T

On Monday, December 21, 2015 at 6:57:39 PM UTC+1, Tero Frondelius wrote:
>
> Hi
> Can you use a macro to produce your twenty nested loops? 

[julia-users] Array view without a single element

[Tomas Lycken]

Is there a way to construct a view into an array that skips a single 
element?

For example, to view v except the element at index p, I can do sub(v, 
[1:p-1;p+1:length(A)]). However, allocating the array with indices is a 
bottleneck in my code. Is there a way or trick to get rid of it?

I’m using this in a recursive function, so there will be subsequent views 
of views of views, etcetera, and I can’t easily refactor it to be a 
(nested) loop instead (because I don’t want to hand-code 20 nested loops…)

// T
​

[julia-users] Survey: what documentation platforms do you use? Are you happy?

[Tomas Lycken]

Over at Interpolations.jl, we've started thinking about restructuring our 
documentation a little. We basically have (or will have) three levels of 
documentation: usage docs, specifying how to use the library; math docs, 
fleshing out the mathematical background to the algorithms and assumptions 
we've used; and devdocs, which describe implementation details that might 
not be obvious just from reading the code (for one thing, the library is 
quite heavy on metaprogramming).

The usage/api docs are slowly being migrated to docstrings, with the hope 
that the users will actually be able to find it :)

I'm looking for input on what tools, formats, platforms etc have worked 
well for the other two types of documentation in other projects, and in 
what form you, as users or contributors to the library, would prefer 
consuming that documentation.

A stong requirement is that it is editable, and preferrably also quite 
readable, as plaintext. For the math docs, it's also important (obviously) 
that the support for rendering equations, matrices etc is good.

Have you used any tools or platforms previously that solve these problems 
well? Do you have recommendations about which ones to avoid?

All suggestions are welcome!

// T

[julia-users] Re: advent of code -- day 3

[Tomas Lycken]

Probably because the arrays will not be equal even if their contents are 
equal (this is true of most mutable types in Julia).

If you try representing a position as a tuple `(0,0)` instead of as an 
array `[0,0]`, you'll find that it works as expected.

// T

On Tuesday, December 15, 2015 at 8:28:36 AM UTC+1, Jan Strube wrote:
>
> That's not a problem at all. In fact that's the very reason why I'm using 
> a Set in the first place.
>
> My question is: Why is the number of entries in the set different when I 
> add Arrays vs. adding ASCIIStrings?
>
>
> On Monday, December 14, 2015 at 10:57:57 PM UTC-8, ele...@gmail.com wrote:
>>
>> I think your problem is that Sets cannot contain duplicate entries, so if 
>> Santa ever passes the same point twice it won't be added.
>>
>> Cheers
>> Lex
>>
>> On Tuesday, December 15, 2015 at 4:23:19 PM UTC+10, Jan Strube wrote:
>>>
>>> I'm trying to learn a bit more Julia by solving the puzzles over on 
>>> http://adventofcode.com
>>> On day 3, the problem is to follow a number of directions and figure out 
>>> how many new places you end up.
>>> http://adventofcode.com/day/3
>>>
>>> I thought I can solve this simply by defining a set of [x y] positions, 
>>> each time adding a new grid position to the set, so I'd end up with a Set{ 
>>> Array{Int64,2}} of the right length.
>>> However, this doesn't work as expected. I get the wrong number (it's too 
>>> low).
>>>
>>> Wrapping each grid position into a string() call, however, gives me the 
>>> right answer. 
>>> The explanation is a bit convoluted. To avoid spoilers I've put the code 
>>> up at https://gist.github.com/jstrube/3d54e15f7d051b72032b
>>>
>>> I don't quite understand this. Is this expected?
>>>
>>>

[julia-users] Re: @time tic() toc()

[Tomas Lycken]

Try 

@time @sync @parallel for i=1:10
...
end

On Thursday, December 10, 2015 at 11:55:58 PM UTC+1, digxx wrote:
>
> Apparently for a parallel loop the @time macro or tic() toc() do not 
> measure the time?!
> I directly get a result for the time but know that the calculation is 
> still ongoing.
> Is the usage like this:
> @time @parallel for i=1:10
>
>
> end
>

[julia-users] Re: Is there a 'smooth.spline' function in Julia as in R? Thank you!

[Tomas Lycken]

Check out the Interpolations package: 
https://github.com/tlycken/Interpolations.jl

It supports quadratic splines today, with a PR for cubic splines which has 
come quite far through the review process. The interface is not quite what 
you're used to from R, but I think you'll find that it's intuitive enough 
anyway, and performant enough to make up for it ;)

If you find it hard to use, do file issues or PRs with suggestions for 
documentation improvements.

// T

On Thursday, December 10, 2015 at 9:48:41 PM UTC+1, meib...@163.com wrote:
>
> Is there a 'smooth.spline' function in Julia as in R? Thank you!
>

[julia-users] Re: JuliaBox not working since upgrade to 0.4.2

[Tomas Lycken]

Hm. I still see the problem...

// T

On Thursday, December 10, 2015 at 9:51:50 PM UTC+1, cdm wrote:
>
>
> appears to be resolved, per a check moments ago (see attached ...).
>
> the 0.5 dev notebook seems to work fine, as well.
>
> anyone aware of a means for choosing which version
> of Julia to run through the JuliaBox Console ...
> doing "julia" at the juser@juliabox prompt launches
> Version 0.3.12 ... ?
>

Re: [julia-users] Re: range bug in 0.4.1?

[Tomas Lycken]

I think it makes perfect sense. If you need a range object where you know 
the start and ending points, you use colon (i.e. `start:step:stop`). If you 
know how many elements you want and what step, but you don't know (or care 
so much) about the stopping point, you use `range`. Just because *you* don't 
have need for it, doesn't mean no-one else does :)

// T

On Monday, December 7, 2015 at 9:53:38 AM UTC+1, Jürgen Bohnert wrote:
>
> I honestly cannot imagine a good application justifying this 'range' 
> function being in the main namespace. What it does is quite 
> counter-intuitive. Or maybe renaming it would be an option?
> Anyway thanks for all your answers guys.
>
> Best,
> Juergen
>
>
> Am Montag, 7. Dezember 2015 04:11:07 UTC+1 schrieb whycrying:
>>
>> Ah. The second argument is the length of the range.
>>
>> And the three arguments's:
>>
>> julia> which(range,(Int,Int,Int))
>>> range{T,S}(a::T, step::S, len::Integer) at range.jl:101
>>>
>>
>> Not so consistent.
>>
>> Well, this is different from Python.
>> Look range in Python(via iPython):
>>
>> In [3]: range?
>>> Docstring:
>>> range(stop) -> range object
>>> range(start, stop[, step]) -> range object
>>>
>>> Return a sequence of numbers from start to stop by step.
>>> Type:  type
>>>
>>
>>
>>
>> -- 
>> https://www.zhangkaizhao.com/
>>
>

[julia-users] Re: what's the easiest way to force a recompilation?

[Tomas Lycken]

Remove the .ji file in .julia/v0.4/.cache; next time you import the 
package, it's re-compiled.

// T

On Friday, December 4, 2015 at 11:50:03 PM UTC+1, Seth wrote:
>
> I'd rather not alter any of the source files. Is there a Pkg command that 
> will recompile? (Pkg.build() doesn't seem to do it).
>

Re: [julia-users] Proposal: NoveltyColors.jl

[Tomas Lycken]

Randy, that was my point exactly :)

// T

On Wednesday, December 2, 2015 at 8:38:28 PM UTC+1, Stefan Karpinski wrote:
>
> Parkinson's law of triviality 
> <https://en.wikipedia.org/wiki/Parkinson%27s_law_of_triviality>
>
> On Wed, Dec 2, 2015 at 2:26 PM, Randy Zwitch <randy@fuqua.duke.edu 
> > wrote:
>
>> On Wednesday, December 2, 2015 at 3:12:36 AM UTC-5, Tomas Lycken wrote:
>>>
>>> Also, the naming discussion here is not *only* on naming; it's also on 
>>> the possibility of including *more things *in the package. I would be 
>>> all for having a package like Palettes.jl, which would include both 
>>> NoveltyColors.jl and other palettes, but that's not in conflict with the 
>>> current package - it's an extension, that might happen tomorrow, in a year, 
>>> or never at all, depending on whether someone actually finds it useful 
>>> enough to implement it.
>>>
>>
>> People are free to extend my package as much as they like, or create a 
>> wrapper package that aggregates all these different packages. Not sure why 
>> that needs a waiting period for my package. 
>>
>> On Wednesday, December 2, 2015 at 1:22:21 PM UTC-5, cormu...@mac.com 
>> wrote:
>>>
>>> I'm using [ColorSchemes.jl](
>>> https://github.com/cormullion/ColorSchemes.jl) for my own purposes, but 
>>> I'm happy to rename it if someone else wants the name.
>>
>>
>> Is this on METADATA? I thought I had checked, but maybe I had missed. If 
>> so, then it makes my package a bit redundant, as I could've submitted a PR 
>> to your package. 
>>
>> On Wednesday, December 2, 2015 at 1:39:09 PM UTC-5, Stefan Karpinski 
>> wrote:
>>>
>>> I'm still pretty unclear on what makes a color scheme a novelty.
>>>
>>>
>> I don't know, it seemed like having a plot based on the colors from the 
>> Grand Budapest Hotel or an outfit Beyonce wore would be a novelty, as 
>> opposed to something like ColorBrewer which explicitly tries to provide 
>> schemes that improve cartography. Since Tim has a majority of the commits 
>> on Color.jl and he thought it was an idea that stood on its own (per my 
>> reading into his approval comment above), I went with it. I really didn't 
>> think it would be this contentious.
>>
>>
>>  
>>
>
>

Re: [julia-users] Proposal: NoveltyColors.jl

[Tomas Lycken]

Also, the naming discussion here is not *only* on naming; it's also on the 
possibility of including *more things *in the package. I would be all for 
having a package like Palettes.jl, which would include both 
NoveltyColors.jl and other palettes, but that's not in conflict with the 
current package - it's an extension, that might happen tomorrow, in a year, 
or never at all, depending on whether someone actually finds it useful 
enough to implement it.

// T

On Tuesday, December 1, 2015 at 4:34:08 PM UTC+1, Randy Zwitch wrote:
>
> Here's my PR, I'm not the one who merged it. However, between November 24 
> and yesterday seems quite long enough for naming; I mean, no one else was 
> really thinking of using NoveltyColors as a name, were they? Naming 
> discussions to me only seem necessary when it's a common name so that 
> someone doesn't squat on a name that's obvious and should be reserved.
>
> https://github.com/JuliaLang/METADATA.jl/pull/4120
>
> Ultimately, I just wanted to add these color palettes to Vega.jl, 
> ColorBrewer.jl already shows that we're willing to have small color 
> packages on METADATA, and I figured other people might want to use these 
> palettes. Moving them inside of Vega.jl would take 30 seconds, so it 
> doesn't really matter to me. Just trying to modularize for the benefit of 
> the community.
>
> On Tuesday, December 1, 2015 at 9:56:41 AM UTC-5, Andreas Lobinger wrote:
>>
>>
>>
>> On Tuesday, December 1, 2015 at 3:32:19 PM UTC+1, Randy Zwitch wrote:
>>>
>>> +1 Tom. 
>>>
>>> It's on METADATA now, so only focus now is to improve the package, the 
>>> name isn't going to change.
>>>
>>
>> I can not track it down right now (i think it was on the Color/Colors,jl 
>> discussion) but wasn't there something like a holding time for METADATA.jl? 
>> Like 48 hours? I cannot really understand, how a package like this with a 
>> special interest flavour and somehow limited code makes it an official 
>> package in a few hours? Without any naming discussion.
>>
>

Re: [julia-users] Best method for inputting logic as a variable in a function

[Tomas Lycken]

Martin Fowler published a (rather long) blog post on the topic of 
refactoring code into data, which I think is a nice addition in this thread 
even though you already have a working solution:

http://martinfowler.com/articles/refactoring-adaptive-model.html

His solution to this problem would have been to build a data model for the 
logic, and then a custom logic engine which can understand and make 
decisions based on the provided data. Perhaps overkill for this use case, 
but it's an interesting approach, IMO.

// T

On Tuesday, December 1, 2015 at 1:22:51 PM UTC+1, NotSoRecentConvert wrote:
>
> It worked quite well. Here is the latest version.
>
> function qcl_valve_plot(Dr::AbstractString,mindate::DateTime,maxdate::
> DateTime,dt::Dates.Period,col::Symbol,valveLogic::Function)
>
> ...
>
> # Valve Zones
> f4 = []
> f4 = find(valveLogic(statuses) .== true)
>
> One of the valveLogic examples, "(V3 & !(V1 & V2)) == true", would now be 
> x -> x.Valve3 & !(x.Valve1 & x.Valve2).
>
> Thanks for the tip.
>

[julia-users] Re: Concatenation without splatting

[Tomas Lycken]

If they’re the same lengths, you could do vcat-then-reshape to get the same 
result as for hcat, but without splatting :)

reshape(fast_vcat(arr_of_arrs), length(arr_of_arrs[1]), length(arr_of_arrs))

// T

On Monday, November 30, 2015 at 3:42:35 PM UTC+1, Cedric St-Jean wrote:

Oh, right, the input vectors are all the same size. It's for a machine 
> learning training set.
>
> I've yet to hit the `hcat(v...)` limit in this particular case. I was just 
> curious to see what the alternatives were, since it seems to be common 
> wisdom that splatting arbitrarily-sized vectors is bad. Thank you for the 
> answers everyone.
>
> Cédric
>
> On Monday, November 30, 2015 at 1:32:42 AM UTC-5, Tomas Lycken wrote:
>>
>> For hcat, what shape would you want the output to have?
>>
>> With your example input, the result of vcat is straightforward:
>>
>> v = [[1],[2,3],[4,5]]
>> vcat(v...)
>> # -> [1, 2, 3, 4, 5]
>>
>> But for hcat, what output would you want? hcat(v...) throws a 
>> DimensionMismatch error stating that vectors must have the same length.
>> //T
>>
>> On Friday, November 27, 2015 at 1:39:34 PM UTC+1, Cedric St-Jean wrote:
>>
>> foldl would work, but it's going to create a ton of temporary arrays.
>>>
>>> None of the proposed efficient solutions work with hcat... I suppose if 
>>> splatting is a problem I should allocate and fill in the array myself.
>>>
>>> On Friday, November 27, 2015 at 6:39:07 AM UTC-5, Glen O wrote:
>>>>
>>>> Any chance that foldl(vcat,arr_of_arr) will do the job?
>>>>
>>>> On Sunday, 22 November 2015 23:04:26 UTC+10, Cedric St-Jean wrote:
>>>>>
>>>>> I have a big vector of vectors. Is there any way to vcat/hcat them 
>>>>> without splatting?
>>>>>
>>>>> arr_of_arr = Vector[[1],[2,3],[4,5]]
>>>>> vcat(arr_of_arr...)
>>>>>
>>>>> I'm asking because splatting big arrays is a performance issue (and 
>>>>> IIRC it blows the stack at some point).
>>>>>
>>>> ​
>>
> ​

[julia-users] Re: Concatenation without splatting

[Tomas Lycken]

For hcat, what shape would you want the output to have?

With your example input, the result of vcat is straightforward:

v = [[1],[2,3],[4,5]]
vcat(v...)
# -> [1, 2, 3, 4, 5]

But for hcat, what output would you want? hcat(v...) throws a 
DimensionMismatch error stating that vectors must have the same length.
//T

On Friday, November 27, 2015 at 1:39:34 PM UTC+1, Cedric St-Jean wrote:

foldl would work, but it's going to create a ton of temporary arrays.
>
> None of the proposed efficient solutions work with hcat... I suppose if 
> splatting is a problem I should allocate and fill in the array myself.
>
> On Friday, November 27, 2015 at 6:39:07 AM UTC-5, Glen O wrote:
>>
>> Any chance that foldl(vcat,arr_of_arr) will do the job?
>>
>> On Sunday, 22 November 2015 23:04:26 UTC+10, Cedric St-Jean wrote:
>>>
>>> I have a big vector of vectors. Is there any way to vcat/hcat them 
>>> without splatting?
>>>
>>> arr_of_arr = Vector[[1],[2,3],[4,5]]
>>> vcat(arr_of_arr...)
>>>
>>> I'm asking because splatting big arrays is a performance issue (and IIRC 
>>> it blows the stack at some point).
>>>
>> ​

[julia-users] [OT?] Will Bret Victor's answer to "What can a technologist do about climate change?" get you to file your first PR? :)

[Tomas Lycken]

A friend just sent me a blog post by Bret Victor, whom I find to be one of 
the more inspiring individuals in the tech community today. The blog post 
is titled

*What can a technologist do about climate change? (a personal view) 
* (
http://worrydream.com/#!/ClimateChange)

It’s long, but it’s well-informed and well-written. After four chapters 
focusing on things that aren’t so specific to the tech-community but 
actually applicable to almost anyone (funding, energy production, energy 
transport and energy consumption), he starts talking about “Tools for 
scientists and engineers”, and (after noting that most tools today aren’t 
great), he says

Here’s an opinion you might not hear much — I feel that one effective 
approach to addressing climate change is contributing to the development of 
Julia .

This makes at least me motivated to push the limits even further here :)

// T
​

Re: [julia-users] Concatenation without splatting

[Tomas Lycken]

That doesn’t quite seem to do what you want, Mauro:

julia> arr_of_arr = Vector{Int}[[1],[2,3],[4,5]]
3-element Array{Array{Int64,1},1}:
 [1]
 [2,3]
 [4,5]

julia> vcat_nosplat(y) = eltype(y[1])[el[1] for el in y]
vcat_nosplat (generic function with 1 method)

julia> vcat_nosplat(arr_of_arr)
3-element Array{Int64,1}:
 1
 2
 4

It’s quite trivial to achieve the desired result with only a few lines of 
code, though:

julia> function vcat_nosplat2(y)
   result = Array(eltype(y[1]), 0)
   sizehint!(result, sum(map(length, y))) #skip if iterating is more 
expensive than reallcoation

   for a in y
   for x in a
   push!(result, x)
   end
   end

   result
   end
vcat_nosplat2 (generic function with 1 method)

julia> vcat_nosplat2(arr_of_arr)
5-element Array{Int64,1}:
 1
 2
 3
 4
 5

// T

On Sunday, November 22, 2015 at 9:12:55 PM UTC+1, Mauro wrote:

In ODE.jl, I've used 
>
> vcat_nosplat(y) = eltype(y[1])[el[1] for el in y] # Does vcat(y...) 
> without the splatting 
>
> I think the eltype might not be needed.  There may be better ways though. 
>
> On Sun, 2015-11-22 at 14:04, Cedric St-Jean  > wrote: 
> > I have a big vector of vectors. Is there any way to vcat/hcat them 
> without 
> > splatting? 
> > 
> > arr_of_arr = Vector[[1],[2,3],[4,5]] 
> > vcat(arr_of_arr...) 
> > 
> > I'm asking because splatting big arrays is a performance issue (and IIRC 
> it 
> > blows the stack at some point). 
>
​

Re: [julia-users] Concatenation without splatting

[Tomas Lycken]

Using append! is definitely an improvement, thanks!

The reason I included the sizehint! call was that this resizes the vector 
*once*, and without requiring to copy any elements, and I assumed those 
saved copies would quickly win over having to iterate the collection an 
extra time to get the total number of elements. I’m glad to be proven wrong 
:)

function vcat_prealloc(y)
result = Array(eltype(y[1]), sum(map(length, y)))

for a in y
append!(result, a)
end
end

function vcat_dynamic(y)
result = Array(eltype(y[1]), 0)
for a in y
append!(result, a)
end
end

arr_of_arrs = Vector{Int}[rand(1:10, rand(1:10)) for _ in 1:10_000]

julia> @benchmark vcat_dynamic(arr_of_arrs)
 Benchmark Results 
 Time per evaluation: 441.11 μs [405.40 μs, 476.82 μs]
Proportion of time in GC: 0.00% [0.00%, 0.00%]
Memory allocated: 1.00 mb
   Number of allocations: 16 allocations
   Number of samples: 100
   Number of evaluations: 100
 Time spent benchmarking: 0.07 s

julia> @benchmark vcat_prealloc(arr_of_arrs)
 Benchmark Results 
 Time per evaluation: 716.85 μs [652.07 μs, 781.62 μs]
Proportion of time in GC: 0.00% [0.00%, 0.00%]
Memory allocated: 933.78 kb
   Number of allocations: 7 allocations
   Number of samples: 100
   Number of evaluations: 100
 Time spent benchmarking: 0.10 s

// T

On Monday, November 23, 2015 at 11:51:25 AM UTC+1, Dan wrote:

Using `append!` instead of `push!` and letting efficient dynamic 
> reallocation of vector do the resizing:
>
> function vcat_nosplat2a(y)
> result = Array(eltype(y[1]), 0)
> for a in y
> append!(result, a)
> end
> result
> end
>
> (@benchmark shows way less allocations and 2-3x time)
> BTW the splat version is just as quick, but perhaps allocation on stack is 
> problematic (anybody check the limit?)
>
> On Monday, November 23, 2015 at 12:07:18 PM UTC+2, Tomas Lycken wrote:
>>
>> That doesn’t quite seem to do what you want, Mauro:
>>
>> julia> arr_of_arr = Vector{Int}[[1],[2,3],[4,5]]
>> 3-element Array{Array{Int64,1},1}:
>>  [1]
>>  [2,3]
>>  [4,5]
>>
>> julia> vcat_nosplat(y) = eltype(y[1])[el[1] for el in y]
>> vcat_nosplat (generic function with 1 method)
>>
>> julia> vcat_nosplat(arr_of_arr)
>> 3-element Array{Int64,1}:
>>  1
>>  2
>>  4
>>
>> It’s quite trivial to achieve the desired result with only a few lines of 
>> code, though:
>>
>> julia> function vcat_nosplat2(y)
>>result = Array(eltype(y[1]), 0)
>>sizehint!(result, sum(map(length, y))) #skip if iterating is more 
>> expensive than reallcoation
>>
>>for a in y
>>for x in a
>>push!(result, x)
>>end
>>end
>>
>>result
>>end
>> vcat_nosplat2 (generic function with 1 method)
>>
>> julia> vcat_nosplat2(arr_of_arr)
>> 5-element Array{Int64,1}:
>>  1
>>  2
>>  3
>>  4
>>  5
>>
>> // T
>>
>> On Sunday, November 22, 2015 at 9:12:55 PM UTC+1, Mauro wrote:
>>
>> In ODE.jl, I've used 
>>>
>>> vcat_nosplat(y) = eltype(y[1])[el[1] for el in y] # Does vcat(y...) 
>>> without the splatting 
>>>
>>> I think the eltype might not be needed.  There may be better ways 
>>> though. 
>>>
>>> On Sun, 2015-11-22 at 14:04, Cedric St-Jean <cedric...@gmail.com> 
>>> wrote: 
>>> > I have a big vector of vectors. Is there any way to vcat/hcat them 
>>> without 
>>> > splatting? 
>>> > 
>>> > arr_of_arr = Vector[[1],[2,3],[4,5]] 
>>> > vcat(arr_of_arr...) 
>>> > 
>>> > I'm asking because splatting big arrays is a performance issue (and 
>>> IIRC it 
>>> > blows the stack at some point). 
>>>
>> ​
>>
> ​

[julia-users] Re: push! performance

[Tomas Lycken]

Making sure that precompilation and gc don’t factor into the result, I get 
quite different timings:

julia> gc(); @time sizehint!(a, 10_000_000);
  0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)

julia> gc(); @time b = zeros(Int, 10_000_000);
  0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)

julia> versioninfo()
Julia Version 0.4.1
Commit cbe1bee* (2015-11-08 10:33 UTC)
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

The gc count on sizehint!() is obviously bogus, but the difference in 
timing is much smaller. I wouldn’t doubt, however, that there is 
performance to be gained from the fact that sizehint! doesn’t have to write 
zeros to all the memory it allocates. My point was rather to illustrate 
*when* allocation occurs, and how *usage* of the array differs depending on 
approach, than the minute implementation details.

// T

On Monday, November 16, 2015 at 4:09:54 PM UTC+1, Seth wrote:

I'm not sure the equivalence is entirely accurate:
>
> julia> a = Vector{Int}()
> 0-element Array{Int64,1}
>
> julia> Base.summarysize(a)
> 0
>
> julia> @time sizehint!(a,10_000_000)
> 0.96 seconds (149 allocations: 76.304 MB)
> 0-element Array{Int64,1}
>
> julia> Base.summarysize(a)
> 0
>
> julia> @time b = zeros(Int,10_000_000);
> 0.037202 seconds (35 allocations: 76.295 MB, 64.13% gc time)
>
> julia> Base.summarysize(b)
> 8000
>
>
>
> On Monday, November 16, 2015 at 1:10:36 AM UTC-8, Tomas Lycken wrote:
>>
>> sizehint! preallocates for you, with comparable cost to calling e.g. 
>> zeroes, but lets you treat the array semantically the same way as a 
>> non-preallocated one (but without the cost for reallocation). Hopefully, 
>> these comments highlight the differences between the various appraches:
>>
>> N = 10_000
>> A = Array(Float64,0) 
>> sizehint!(A, 10_000) # this preallocates memory for 10k elements
>> B = Array(Float64,0)
>> C = zeros(10_000) # this also preallocates memory for 10k elements
>>
>> # now, A and C are pre-allocated, while B is not# however, A and B are 
>> semantically equivalent (0-length) vectors,
>> # while C is already of length 10 000:
>> println(length(A)) # 0
>> println(length(B)) # 0
>> println(length(C)) # 1
>>
>> for i in 1:10_000
>>push!(A, i) # no reallocation happens here, because we did it with 
>> sizehint!
>>push!(B, i) # this will re-allocate B every now and then
>>C[i] = i # can't use push! here, but must manually track index instead
>> end
>>
>> I don't know what `dynamic` does in this context, and I can't find it in 
>> the docs, so can't help you there :)
>>
>> // T
>>
>> On Monday, November 16, 2015 at 2:07:13 AM UTC+1, Seth wrote:
>>
>> What happens if you use sizehint!() with dynamic()?
>>>
>>> On Sunday, November 15, 2015 at 3:35:45 PM UTC-8, Steven G. Johnson 
>>> wrote:
>>>>
>>>> function prealloc(n)
>>>> a = Array(Int, n)
>>>> for i = 1:n
>>>> a[i] = i
>>>> end
>>>> return a
>>>> end
>>>> function dynamic(n)
>>>> a = Int[]
>>>> for i = 1:n
>>>> push!(a, i)
>>>> end
>>>> return a
>>>> end
>>>> @time prealloc(10^7);
>>>> @time dynamic(10^7);
>>>>
>>>>
>>>> On my machine, the preallocated version is 2.5–3x faster.  A 
>>>> significant but not overwhelming margin.
>>>>
>>> ​
>>
> ​

[julia-users] Re: Progress meter for parallel workers

[Tomas Lycken]

There has been some discussion about this, 
see https://github.com/timholy/ProgressMeter.jl/issues/9 
and https://github.com/timholy/ProgressMeter.jl/issues/32

// T

On Monday, November 16, 2015 at 1:53:22 PM UTC+1, bernhard wrote:
>
> related to this I would welcome if it were possible to show the progress 
> of a pmap() statement.
> It is easy, to have each worker display the instance number which is being 
> processed (say if pmap goes over a range 1:n). But I do not know how to 
> show progress and estimated time left
>
> Am Sonntag, 15. November 2015 16:35:24 UTC+1 schrieb Jason Eckstein:
>>
>>
>>
>> I was playing around with the ProgressMeter.jl package by Tim Holy and it 
>> works great for a single threaded loop.  I was wondering if there's any way 
>> to run several loops on multiple workers and have a set of progress bars 
>> update in real time.  When I tried this with Tim's package it just said 
>> Progress: 100% Time:  after each was complete but nothing during 
>> intermediate progress.  What I had in mind was a display like
>> Worker 1:  Progress: X%...
>> Worker 2: Progress: Y% 
>> that update in real time but have multiple lines to represent the 
>> different workers.
>>
>

[julia-users] Re: push! performance

[Tomas Lycken]

Yeah, that makes it even more clear that they’re very similar:

julia> gc(); @time for i=1:1_000; sizehint!(Vector{Int}(0), 1_000); end
  0.000186 seconds (2.00 k allocations: 7.721 MB)

julia> gc(); @time for i=1:1_000; Vector{Int}(1_000); end
  0.000181 seconds (1000 allocations: 7.706 MB)

(broken into smaller chunks, because the timings started to get awfully 
close to the resolution of the clock, and the gc percentage count got 
really borked…)

// T

On Monday, November 16, 2015 at 6:26:04 PM UTC+1, Steven G. Johnson wrote:


>
> On Monday, November 16, 2015 at 11:24:20 AM UTC-5, Tomas Lycken wrote:
>>
>> Making sure that precompilation and gc don’t factor into the result, I 
>> get quite different timings:
>>
>> julia> gc(); @time sizehint!(a, 10_000_000);
>>   0.001493 seconds (46 allocations: 76.296 MB, 304.61% gc time)
>>
>> julia> gc(); @time b = zeros(Int, 10_000_000);
>>   0.021997 seconds (38 allocations: 76.296 MB, 0.70% gc time)
>>
>>
> Compare to Array(Int,  10_000_000), which allocates an uninitialized array.
>
>> ​
>>>>
>>> ​
>>
> ​

[julia-users] Re: push! performance

[Tomas Lycken]

sizehint! preallocates for you, with comparable cost to calling e.g. zeroes, 
but lets you treat the array semantically the same way as a 
non-preallocated one (but without the cost for reallocation). Hopefully, 
these comments highlight the differences between the various appraches:

N = 10_000
A = Array(Float64,0) 
sizehint!(A, 10_000) # this preallocates memory for 10k elements
B = Array(Float64,0)
C = zeros(10_000) # this also preallocates memory for 10k elements

# now, A and C are pre-allocated, while B is not# however, A and B are 
semantically equivalent (0-length) vectors,
# while C is already of length 10 000:
println(length(A)) # 0
println(length(B)) # 0
println(length(C)) # 1

for i in 1:10_000
   push!(A, i) # no reallocation happens here, because we did it with sizehint!
   push!(B, i) # this will re-allocate B every now and then
   C[i] = i # can't use push! here, but must manually track index instead
end

I don't know what `dynamic` does in this context, and I can't find it in 
the docs, so can't help you there :)

// T

On Monday, November 16, 2015 at 2:07:13 AM UTC+1, Seth wrote:

What happens if you use sizehint!() with dynamic()?
>
> On Sunday, November 15, 2015 at 3:35:45 PM UTC-8, Steven G. Johnson wrote:
>>
>> function prealloc(n)
>> a = Array(Int, n)
>> for i = 1:n
>> a[i] = i
>> end
>> return a
>> end
>> function dynamic(n)
>> a = Int[]
>> for i = 1:n
>> push!(a, i)
>> end
>> return a
>> end
>> @time prealloc(10^7);
>> @time dynamic(10^7);
>>
>>
>> On my machine, the preallocated version is 2.5–3x faster.  A significant 
>> but not overwhelming margin.
>>
> ​

Re: [julia-users] Re: Large Data Sets in Julia

[Tomas Lycken]

Everything after the semicolon is keyword arguments, and will dispatch to 
the same method as if they are left out. Thus, the documentation for 
SharedArray(T, 
dims; init=false, pids=[]) is valid for SharedArray(T, dims) too, and the 
values of init and pids will be the ones given in the signature.

// T

On Monday, November 9, 2015 at 9:43:17 PM UTC+1, John Brock wrote:

It looks like SharedArray(filename, T, dims) isn't documented, 
> but SharedArray(T, dims; init=false, pids=Int[]) is. What's the difference? 
>
> On Friday, November 6, 2015 at 2:21:01 AM UTC-8, Tim Holy wrote:
>>
>> Not sure if it's as high-level as you're hoping for, but julia has great 
>> support for arrays that are much bigger than memory. See Mmap.mmap and 
>> SharedArray(filename, T, dims). 
>>
>> --Tim 
>>
>> On Thursday, November 05, 2015 06:33:52 PM André Lage wrote: 
>> > hi Viral, 
>> > 
>> > Do you have any news on this? 
>> > 
>> > André Lage. 
>> > 
>> > On Wednesday, July 3, 2013 at 5:12:06 AM UTC-3, Viral Shah wrote: 
>> > > Hi all, 
>> > > 
>> > > I am cross-posting my reply to julia-stats and julia-users as there 
>> was a 
>> > > separate post on large logistic regressions on julia-users too. 
>> > > 
>> > > Just as these questions came up, Tanmay and I have been chatting 
>> about a 
>> > > general framework for working on problems that are too large to fit 
>> in 
>> > > memory, or need parallelism for performance. The idea is simple and 
>> based 
>> > > on providing a convenient and generic way to break up a problem into 
>> > > subproblems, each of which can then be scheduled to run anywhere. To 
>> start 
>> > > with, we will implement a map and mapreduce using this, and we hope 
>> that 
>> > > it 
>> > > should be able to handle large files sequentially, distributed data 
>> > > in-memory, and distributed filesystems within the same framework. Of 
>> > > course, this all sounds too good to be true. We are trying out a 
>> simple 
>> > > implementation, and if early results are promising, we can have a 
>> detailed 
>> > > discussion on API design and implementation. 
>> > > 
>> > > Doug, I would love to see if we can use some of this work to 
>> parallelize 
>> > > GLM at a higher level than using remotecall and fetch. 
>> > > 
>> > > -viral 
>> > > 
>> > > On Tuesday, July 2, 2013 11:10:35 PM UTC+5:30, Douglas Bates wrote: 
>> > >> On Tuesday, July 2, 2013 6:26:33 AM UTC-5, Raj DG wrote: 
>> > >>> Hi all, 
>> > >>> 
>> > >>> I am a regular user of R and also use it for handling very large 
>> data 
>> > >>> sets (~ 50 GB). We have enough RAM to fit all that data into memory 
>> for 
>> > >>> processing, so don't really need to do anything additional to 
>> chunk, 
>> > >>> etc. 
>> > >>> 
>> > >>> I wanted to get an idea of whether anyone has, in practice, 
>> performed 
>> > >>> analysis on large data sets using Julia. Use cases range from 
>> performing 
>> > >>> Cox Regression on ~ 40 million rows and over 10 independent 
>> variables to 
>> > >>> simple statistical analysis using T-Tests, etc. Also, how does the 
>> > >>> timings 
>> > >>> for operations like logistic regressions compare to Julia ? Are 
>> there 
>> > >>> any 
>> > >>> libraries/packages that can perform Cox, Poisson (Negative 
>> Binomial), 
>> > >>> and 
>> > >>> other regression types ? 
>> > >>> 
>> > >>> The benchmarks for Julia look promising, but in today's age of the 
>> "big 
>> > >>> data", it seems that the capability of handling large data is a 
>> > >>> pre-requisite to the future success of any new platform or 
>> language. 
>> > >>> Looking forward to your feedback, 
>> > >> 
>> > >> I think the potential for working with large data sets in Julia is 
>> better 
>> > >> than that in R.  Among other things Julia allows for memory-mapped 
>> files 
>> > >> and for distributed arrays, both of which have great potential. 
>> > >> 
>> > >> I have been working with some Biostatisticians on a prototype 
>> package for 
>> > >> working with snp data of the sort generated in genome-wide 
>> association 
>> > >> studies.  Current data sizes can be information on tens of thousands 
>> of 
>> > >> individuals (rows) for over a million snp positions (columns).  The 
>> > >> nature 
>> > >> of the data is such that each position provides one of four 
>> potential 
>> > >> values, including a missing value.  A compact storage format using 2 
>> bits 
>> > >> per position is widely used for such data.  We are able to read and 
>> > >> process 
>> > >> such a large array in a few seconds using memory-mapped files in 
>> Julia. 
>> > >> 
>> > >>  The amazing thing is that the code is pure Julia.  When I write in 
>> R I 
>> > >>  am 
>> > >> 
>> > >> always conscious of the bottlenecks and the need to write C or C++ 
>> code 
>> > >> for 
>> > >> those places.  I haven't encountered cases where I need to write new 
>> code 
>> > >> in a compiled language to speed up a Julia function.  I have 
>> interfaced 
>> > >> to 
>> > >> existing 

[julia-users] Re: Type stability with eigs

[Tomas Lycken]

This actually seems to be a type instability issue in Base.eigs:

julia> @code_warntype eigs(A)
Variables:
  A::SparseMatrixCSC{Float64,Int64}

Body:
  begin
  $(Expr(:line, 50, symbol("linalg/arnoldi.jl"), symbol("")))
  GenSym(8) = 
(top(ccall))(:jl_alloc_array_1d,(top(apply_type))(Base.Array,Any,1)::Type{Array{Any,1}},(top(svec))(Base.Any,Base.Int)::SimpleVector,Array{Any,1},0,0,0)::Array{Any,1}
  GenSym(9) = GenSym(8)
  return 
(Base.LinAlg.__eigs#221__)(GenSym(9),A::SparseMatrixCSC{Float64,Int64})::Tuple
  end::Tuple

julia> @code_warntype eigs(Symmetric(A))
Variables:
  A::Symmetric{Float64,SparseMatrixCSC{Float64,Int64}}

Body:
  begin
  $(Expr(:line, 50, symbol("linalg/arnoldi.jl"), symbol("")))
  GenSym(8) = 
(top(ccall))(:jl_alloc_array_1d,(top(apply_type))(Base.Array,Any,1)::Type{Array{Any,1}},(top(svec))(Base.Any,Base.Int)::SimpleVector,Array{Any,1},0,0,0)::Array{Any,1}
  GenSym(9) = GenSym(8)
  return 
(Base.LinAlg.__eigs#221__)(GenSym(9),A::Symmetric{Float64,SparseMatrixCSC{Float64,Int64}})::Tuple
  end::Tuple

I can’t find anything on the issue tracker about it, so it might be a good 
idea to file a new issue .

// t

On Saturday, November 7, 2015 at 5:25:30 PM UTC+1, David Gleich wrote:

I'm trying to get a type-stable output from a symmetric sparse eigenvalue 
> problem.
>
> Any ideas why neither of these ideas work to give me a deterministic 
> return of Array{Float64,1} ?
>
> n = 500; A = sparse(1:n-1,2:n,1.,n,n); A = A + A'
> @show issym(A)
> function myeig1(A::SparseMatrixCSC{Float64,Int64})
>d,V = eigs(A;tol=1e-4)
>return d
> end
>
> function myeig2(A::SparseMatrixCSC{Float64,Int64})
>d,V = eigs(Symmetric(A);tol=1e-4)
>return d
> end
>
> @code_warntype myeig1(A)
> @code_warntype myeig2(A)
>
> Thanks,
> David Gleich
>
​

[julia-users] Re: DataFrame to JSON

[Tomas Lycken]

I’d try to use the built-in tools for iterating and transforming stuff to 
make the code a little terser and easier to read:

function df2json(df)
io = IOBuffer()
write(io, "[\n")
write(io, join(map(row -> "{\"A\": $(row[:A]), \"B\": \"$(row[:B])\"}", 
eachrow(d)), ",\n"))
write(io, "\n]\n")
json = takebuf_string(io)
close(io)
json
end

The keys here is to use the map and eachrow functions to get rid of the for 
loops, and to build each item with string interpolation, which makes it 
easier to see what the result will be.
Note that I end the entire message with a newline; I think this is good 
practice in any string buffer protocol.

Next, I’d split the handling of the IO buffer away from the rest:

function df2json(df)
io = IOBuffer()
df2json(io, df)
json = takebuf_string(io)
close(io)
json
end

function df2json(io::IO, df)
write(io, "[\n")
write(io, join(map(row -> "{\"A\": $(row[:A]), \"B\": \"$(row[:B])\"}", 
eachrow(d)), ",\n"))
write(io, "\n]\n")
end

I’d also make pretty-printing optional, to make it possible to write the 
JSON as compact as possible too:

function df2json(df; pretty_print::Bool=false)
io = IOBuffer()
df2json(io, df; pretty_print=pretty_print)
json = takebuf_string(io)
close(io)
json
end

function df2json(io::IO, df; pretty_print::Bool=false)
write(io, "[")
pretty_print && write(io, "\n")
separator = "," * (pretty_print ? "\n\t" : "")
space = pretty_print ? " " : ""
write(io, join(map(row -> 
"{\"A\":$space$(row[:A]),$space\"B\":$space\"$(row[:B])\"}", eachrow(d)), 
separator))
pretty_print && write(io, "\n")
write(io, "]")
write(io, "\n")
end

Now, this allows you to print the message either for human reading:

julia> d = DataFrame(A=1:10, B=map(x -> x > 0.5 ? "F" : "M", rand(10)))
10x2 DataFrames.DataFrame
| Row | A  | B   |
|-||-|
| 1   | 1  | "M" |
| 2   | 2  | "M" |
| 3   | 3  | "M" |
| 4   | 4  | "M" |
| 5   | 5  | "F" |
| 6   | 6  | "F" |
| 7   | 7  | "F" |
| 8   | 8  | "M" |
| 9   | 9  | "F" |
| 10  | 10 | "F" |

julia> println(df2json(d; pretty_print=true));
[
{"A": 1, "B": "M"},
{"A": 2, "B": "M"},
{"A": 3, "B": "M"},
{"A": 4, "B": "M"},
{"A": 5, "B": "F"},
{"A": 6, "B": "F"},
{"A": 7, "B": "F"},
{"A": 8, "B": "M"},
{"A": 9, "B": "F"},
{"A": 10, "B": "F"}
]

or for minimizing network overhead:

julia> println(df2json(d))
[{"A":1,"B":"M"},{"A":2,"B":"M"},{"A":3,"B":"M"},{"A":4,"B":"M"},{"A":5,"B":"F"},{"A":6,"B":"F"},{"A":7,"B":"F"},{"A":8,
"B":"M"},{"A":9,"B":"F"},{"A":10,"B":"F"}]

// T

On Monday, November 9, 2015 at 6:12:31 AM UTC+1, Eric Forgy wrote:

This is embarrassing since I'm just learning, but in the interest of 
> getting feedback and improving my Julia coding skills, here is what I did:
>
> function df2json(df::DataFrame)
>
> nrow,ncol = size(df)
>
> io = IOBuffer();
> write(io,"[\n")
> for irow = 1:nrow
>   irow == nrow ? eor = "" : eor = ","
>   write(io,"{")
>   for icol = 1:ncol
> icol == ncol ? eoe = "" : eoe = ","
> sym = names(df)[icol]
> name = string(sym)
> if isa(value,Number)
>   write(io,"\""*name*"\":"*string(df[irow,sym])*eoe)
> else
>   write(io,"\""*name*"\":\""*df[irow,sym]*"\""*eoe)
> end
>   end
>   write(io,"}"*eor*"\n")
> end
> write(io,"]\n")
>
> json = takebuf_string(io)
> close(io)
> json
> end
>
>
> Any thoughts/suggestions? Thank you.
>
​

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Tomas Lycken]

My point is that what we *really* want here is the ability to test the new 
behavior while the old behavior is still the released version, or to 
opt-out of the change altogether - all the suggestions in this thread about 
how to handle this transition are trying to solve that problem. And 
solutions already exists - to pre-test, just test the package against the 
Julia master as soon as the change merges (and this will be done by Pkg 
evaluator, semi-automatically filing issues in case errors occur), and to 
opt-out, just pin your code to depend on julia 0.4.

The way I see it, the need for soft transitioning is not as high in Julia 
as it is in Python - Julia is a fast-moving target, and currently in 
pre-release (think of all the 0.x releases as alpha releases of the real 
language, if you will). IMO, it’s reasonable to expect breaking changes now 
and then, even major ones, even ones that may lead to subtle and 
hard-to-find bugs, and it’s reasonable to expect users and package 
developers to either put in some extra effort to keep code up-to-date with 
the latest and greatest, or pin down dependencies to a version one knows 
works (no-one will force you to upgrade to 0.5, if you think 0.4 works 
better for you…). Having to put that effort in is the price you pay for 
using a system that hasn't reached 1.0 status yet. At least to me, Julia is 
awesome enough to make it more than worth it :)

There were a couple of big breakages during the 0.4 cycle (this slide in 
Iain Dunning’s talk from JuliaCon 2015 
<https://youtu.be/pYYmnBma9qk?list=PLP8iPy9hna6Sdx4soiGrSefrmOPdUWixM=417> 
gives a good overview of the big ones) and the community were quite quick 
to fix the major problems that occurred then, so that now when 0.4 was 
released, the actual breakage from those changes was relatively minor. As I 
see it, this transitioning model is good enough for Julia at the moment - 
if/when we want to make breaking changes between 1.x and 2.0, that’s a 
different story.

// T

On Tuesday, November 3, 2015 at 9:28:43 AM UTC+1, Kristoffer Carlsson wrote:

I don't see how this enables "extreme performance gains". As have been 
> said, you can already now use slice/sub in base so the only difference is 
> if a slice is default or not for the [:] syntax. I do realize that syntax 
> matters and no one is going to write slice(A, :, 1) everywhere instead of 
> A[:,1].
>
> However, I am a bit wary about it because the vast majority of the hard to 
> track down bugs I've had has been due to some unexpected shared state. 
> Wouldn't it be best if you could opt in to use the [:] syntax as slices 
> somehow and have the default stay as it is?
>
>
> On Tuesday, November 3, 2015 at 8:30:04 AM UTC+1, Tomas Lycken wrote:
>>
>> There’s no way this transition is going to be smooth without work from 
>> package devs to ensure compatibility anyway, but recent history (e.g. the 
>> Tuplocalypse) shows that packages recover quite rapidly due to the 
>> automated testing done by PkgEval.jl. Thus, packages with decent test 
>> suites will probably get a heads-up that this change is breaking for them 
>> long before 0.5.0 is released.
>>
>> Given that these changes potentially allow quite extreme performance 
>> gains in many applications, I find it unlikely that package devs won’t want 
>> to spend some time fixing any bugs this behavior change introduces, but for 
>> those that just want to pin their package down before fixing it, they can 
>> say julia 0.4 0.4 or similar in their REQUIRE.jl, and not even allow 
>> installation on 0.5 (until they fix the problems, change the REQUIRE.jl and 
>> tag a new version). Note that many (most?) packages won’t even need this - 
>> package testing on the master branch will reveal to devs whether a package 
>> even needs touch-up or not.
>>
>> I used from __future__ import print at the top of my python scripts for 
>> a long time, so I’m entirely in favor of something like that. However, I 
>> guess the behavior needs to be implemented and done in the master branch of 
>> Julia before we know exactly how to mimic it in a package (if it’s even 
>> possible).
>>
>> // T
>>
>> On Tuesday, November 3, 2015 at 7:41:17 AM UTC+1, Christoph Ortner wrote:
>>
>> Why not do this via Compat.jl ?
>>> C
>>>
>>> On Tuesday, 3 November 2015 04:28:44 UTC, David Anthoff wrote:
>>>>
>>>> No, Compat generall ports the new syntax back to old versions. What I'm 
>>>> suggesting is quite different, namely that in julia 0.5 if you just use 
>>>> `[]` for slicing without anything else, you generate a warning, and then 
>>>> you have to opt-in to the NEW syntax. 
>>>>
>>>> 

[julia-users] Re: help on a MethodError

[Tomas Lycken]

Most likely, you’ve imported or defined different versions of the type, 
and/or you have an old definition laying around in the REPL session 
somewhere. convert tries to convert an instance of the old version of the 
type, and fails, because the convert is defined for a newer version. (You 
can think of this as there existing two different types with the same name, 
and the convert method you want is defined for the wrong one…)

Try restarting the REPL, and the problem will probably go away.

// T

On Tuesday, November 3, 2015 at 9:24:57 AM UTC+1, Lanting Guo wrote:

I want to convert a self defined type COSMIC.OrderedSampleMatrix to 
> Array{ASCIIString,2}, but the repl displays a MethodError. But run in the 
> script, everything is OK. I am confused about this.
>
>
> 
>
> ​

Re: [julia-users] Re: Re: are array slices views in 0.4?

[Tomas Lycken]

There’s no way this transition is going to be smooth without work from 
package devs to ensure compatibility anyway, but recent history (e.g. the 
Tuplocalypse) shows that packages recover quite rapidly due to the 
automated testing done by PkgEval.jl. Thus, packages with decent test 
suites will probably get a heads-up that this change is breaking for them 
long before 0.5.0 is released.

Given that these changes potentially allow quite extreme performance gains 
in many applications, I find it unlikely that package devs won’t want to 
spend some time fixing any bugs this behavior change introduces, but for 
those that just want to pin their package down before fixing it, they can 
say julia 0.4 0.4 or similar in their REQUIRE.jl, and not even allow 
installation on 0.5 (until they fix the problems, change the REQUIRE.jl and 
tag a new version). Note that many (most?) packages won’t even need this - 
package testing on the master branch will reveal to devs whether a package 
even needs touch-up or not.

I used from __future__ import print at the top of my python scripts for a 
long time, so I’m entirely in favor of something like that. However, I 
guess the behavior needs to be implemented and done in the master branch of 
Julia before we know exactly how to mimic it in a package (if it’s even 
possible).

// T

On Tuesday, November 3, 2015 at 7:41:17 AM UTC+1, Christoph Ortner wrote:

Why not do this via Compat.jl ?
> C
>
> On Tuesday, 3 November 2015 04:28:44 UTC, David Anthoff wrote:
>>
>> No, Compat generall ports the new syntax back to old versions. What I'm 
>> suggesting is quite different, namely that in julia 0.5 if you just use 
>> `[]` for slicing without anything else, you generate a warning, and then 
>> you have to opt-in to the NEW syntax. 
>>
>> > -Original Message- 
>> > From: julia...@googlegroups.com [mailto:julia- 
>> > us...@googlegroups.com] On Behalf Of Eric Forgy 
>> > Sent: Monday, November 2, 2015 3:30 PM 
>> > To: julia-users  
>> > Subject: RE: [julia-users] Re: Re: are array slices views in 0.4? 
>> > 
>> > Hi David, 
>> > 
>> > I'm not an expert, i.e. I've never used it, but your idea "using 
>> OldArrays" 
>> > sounds exactly like the purpose of Compat.jl, which I believe is the 
>> > compatibility module referred to. 
>> > 
>> > Have a look and good luck: 
>> > 
>> > https://github.com/JuliaLang/Compat.jl 
>>
> ​

Re: [julia-users] Re: typealias vs. const

[Tomas Lycken]

Thanks, the results were interesting enough :)

julia> expand(:(typealias A B))
:(begin
const A
A = B
return B
end)

julia> expand(:(typealias A{T} B{T,1}))
:((AST(:($(Expr(:lambda, Any[:T], Any[Any[Any[:T,:Any,0]],Any[],1,Any[]], 
:(begin
const A
GenSym(0) = 
(top(TypeConstructor))((top(svec))(T),(top(apply_type))(Main.B,T,1))
Main.A = GenSym(0)
return GenSym(0)
end))((top(TypeVar))(:T,top(Any),true)))

Conclusion: Non-parametric type aliases are simple. Parametric type aliases 
are not :)

// T

On Friday, October 30, 2015 at 7:31:57 PM UTC+1, Yichao Yu wrote:

On Fri, Oct 30, 2015 at 1:34 PM, Tomas Lycken <tomas@gmail.com 
> > wrote: 
> >> `const A = T` is what `typealias A T` lowered to when it doesn't have 
> type 
> >> parameters 
> > 
> > I wanted to inspect the output of this lowering pass, but 
> `@code_lowered` 
> > was apparently not what I was looking for (at least, `@code_lowered 
> > :(typealias Foo Int)` didn't work...). 
> > 
> > However, I understand your statement as follows: 
> > 
> > `typealias A T` is equivalent to `const A = T`, while `typealias A{S} 
> T{S}` 
> > does something more involved? What does `typealias A{S} T{Foo,S}` do? I 
> > assume these are not possible using `const  = ` constructs, 
> right? 
>
> right. Try `expand(:(typealias A B))` 
>
> > 
> > // T 
> > 
> > On Friday, October 30, 2015 at 1:45:37 PM UTC+1, Yichao Yu wrote: 
> >> 
> >> On Fri, Oct 30, 2015 at 8:29 AM, FANG Colin <coli...@gmail.com> wrote: 
> >> > I have got the same confusion. Any ideas? 
> >> > 
> >> > I have even seen usage of A = T (no const) 
> >> > (http://docs.julialang.org/en/release-0.4/manual/types/#type-unions), 
> is 
> >> > it 
> >> > supposed to be bad (slow) because it is a global variable? 
> >> > 
> >> 
> >> no const will have performance issue since it's a global. 
> >> `const A = T` is what `typealias A T` lowered to when it doesn't have 
> >> type parameters. In this case, it's just a matter of style. Being 
> >> consistent within a file is probably better but you can pick whichever 
> >> you prefer. 
> >> 
> >> > 
> >> > 
> >> > On Thursday, November 6, 2014 at 3:38:41 PM UTC, Steven G. Johnson 
> >> > wrote: 
> >> >> 
> >> >> Given a type T, what is the difference in practice between 
> >> >>  typealias A T 
> >> >> and 
> >> >>  const A = T 
> >> >> ? 
> >> >> 
> >> >> There seems to be some disagreement over which one to use (e.g. 
> >> >> 
> >> >> 
> https://github.com/JuliaLang/Compat.jl/commit/8211e38ac7d8448298a2bb3ab36d6f0b6398b577).
>  
>
> >> >> 
> >> >> My impression is that there is no difference, and that the only 
> >> >> advantage 
> >> >> of a typealias is that it can be parameterized.  Is that right? 
> >> >> 
> >> >> If they are equivalent, what is the Julian style?  Even in Julia 
> Base 
> >> >> it 
> >> >> doesn't seem to be entirely consistent. 
>
​

Re: [julia-users] Re: typealias vs. const

[Tomas Lycken]

> `const A = T` is what `typealias A T` lowered to when it doesn't 
have type parameters

I wanted to inspect the output of this lowering pass, but `@code_lowered` 
was apparently not what I was looking for (at least, `@code_lowered 
:(typealias Foo Int)` didn't work...).

However, I understand your statement as follows:

`typealias A T` is equivalent to `const A = T`, while `typealias A{S} T{S}` 
does something more involved? What does `typealias A{S} T{Foo,S}` do? I 
assume these are not possible using `const  = ` constructs, right?

// T 

On Friday, October 30, 2015 at 1:45:37 PM UTC+1, Yichao Yu wrote:
>
> On Fri, Oct 30, 2015 at 8:29 AM, FANG Colin  > wrote: 
> > I have got the same confusion. Any ideas? 
> > 
> > I have even seen usage of A = T (no const) 
> > (http://docs.julialang.org/en/release-0.4/manual/types/#type-unions), 
> is it 
> > supposed to be bad (slow) because it is a global variable? 
> > 
>
> no const will have performance issue since it's a global. 
> `const A = T` is what `typealias A T` lowered to when it doesn't have 
> type parameters. In this case, it's just a matter of style. Being 
> consistent within a file is probably better but you can pick whichever 
> you prefer. 
>
> > 
> > 
> > On Thursday, November 6, 2014 at 3:38:41 PM UTC, Steven G. Johnson 
> wrote: 
> >> 
> >> Given a type T, what is the difference in practice between 
> >>  typealias A T 
> >> and 
> >>  const A = T 
> >> ? 
> >> 
> >> There seems to be some disagreement over which one to use (e.g. 
> >> 
> https://github.com/JuliaLang/Compat.jl/commit/8211e38ac7d8448298a2bb3ab36d6f0b6398b577).
>  
>
> >> 
> >> My impression is that there is no difference, and that the only 
> advantage 
> >> of a typealias is that it can be parameterized.  Is that right? 
> >> 
> >> If they are equivalent, what is the Julian style?  Even in Julia Base 
> it 
> >> doesn't seem to be entirely consistent. 
>

Re: [julia-users] Re: Order of multiple for-loops

[Tomas Lycken]

in my experience practical applications favor row ordering

The keyword there is that it was *your* experience. The thing about 
something as central to number crunching as organizing memory layouts in 
row- or column major orderings, is that there are more practical 
applications than any of us can imagine - in fact, most likely we couldn’t 
come up with all possible cases where one is significantly better than the 
other even if we stopped discussing Julia and just discussed that for a 
year.

For some applications column major is better, but for others row major 
rocks. We have to choose, and even if the choice was once quite arbitrary, 
the gain from switching is so extremely low it’s ridiculous to even 
consider it. Thus, I don’t think it’s fruitful to discuss here.

However, Julia is an incredibly expressive language, and you could quite 
easily write a macro that re-orders the indices for you:

julia> macro rowmajor(expr)
   @assert expr.head == :ref

   Expr(:ref, expr.args[1], expr.args[end:-1:2]...)
   end

julia> macroexpand(:(@rowmajor A[i,j]))
:(A[j,i])

Of course, this only works on one indexing expression at a time, but it can 
be extended to work recursively through an expression tree and rewrite all 
array indexing expressions it finds, allowing usage as

@rowmajor begin
# formulate row-major algorithm here
end

No need to change the language - just bend it to do what you need :)

// T
On Friday, October 30, 2015 at 5:25:36 PM UTC+1, Tom Breloff wrote:

Preference for row ordering is more related to the way people view tabular 
> data, I think.  For many applications/data, there are many more rows than 
> columns (think of database tables or csv files), and it's slightly 
> unnatural to read those into a transposed data structure for analysis.  
> Putting a time series into a TxN matrix is natural (that's likely how the 
> data is stored), but inefficient if data access is sequentially through 
> time.  Without a "TransposeView" or similar, we're forced to make a choice 
> between poor performance or an unintuitive representation of the data.
>
> I can appreciate that matrix operations could be more natural with column 
> ordering, but in my experience practical applications favor row ordering.
>
> On Fri, Oct 30, 2015 at 11:44 AM, John Gibson  > wrote:
>
>> Agreed w Glenn H here. "math being column major" is because the range of 
>> a matrix being the span of its columns, and consequently most linear 
>> algebra algorithms are naturally expressed as operations on the columns of 
>> the matrix, for example QR decomp via Gramm-Schmidt or Householder, LU 
>> without pivoting, all Krylov subspace methods. An exception would be LU 
>> with full or partial row pivoting. 
>>
>> I think preference for row-ordering comes from the fact that the textbook 
>> presentation of matrix-vector multiply is given as computing y(i)= sum_j 
>> A(i,j) x(j) for each value of i. Ordering the operations that way would 
>> make row-ordering of A cache-friendly. But if instead you understand 
>> mat-vec mult as forming a linear combination of the columns of A, and you 
>> do the computation via y = sum_j A(:,j) x(j), column-ordering is 
>> cache-friendly. And it's the latter version that generalizes into all the 
>> important linear algebra algorithms.
>>
>> John
>>
>>
>> On Friday, October 30, 2015 at 10:46:36 AM UTC-4, Glen H wrote:
>>>
>>>
>>> On Thursday, October 29, 2015 at 1:24:23 PM UTC-4, Stefan Karpinski 
>>> wrote:

 Yes, this is an unfortunate consequence of mathematics being 
 column-major – oh how I wish it weren't so. The storage order is actually 
 largely irrelevant, the whole issue stems from the fact that the element 
 in 
 the ith row and the jth column of a matrix is indexes as A[i,j]. If it 
 were 
 A[j,i] then these would agree (and many things would be simpler). I like 
 your explanation of "an index closer to the expression to be evaluated 
 runs faster" – that's a really good way to remember/explain it.


>>> To help understand, is "math being column major" referring to matrix 
>>> operations in math textbooks are done by columns?  For example:
>>>
>>>
>>> http://eli.thegreenplace.net/2015/visualizing-matrix-multiplication-as-a-linear-combination/
>>>
>>> While the order is by convention (eg not that is has to be that way), 
>>> this is how people are taught.
>>>
>>> Glen
>>>
>>
> ​

Re: [julia-users] Re: For loop = or in?

[Tomas Lycken]

I read 1:n that way. And even if I didn't, I prefer to think about a looping 
construct as "do this for each element in that collection" - and regardless of 
how I pronounce 1:n, I see it as a collection since it inherits AbstractArray. 

But I can also see how other mental models work for you. There's a lot of talk 
in this thread along the lines of "my mental model of this construct is better 
than yours, and therefore my syntax preference is better than yours." I find 
this ridiculously inconstructive - bordering on children arguing in the 
sandbox... To me, this just indicates that both mental models are valid that 
both syntaxes should, at least for the time being, stay. The fact that no clear 
winner can be found by counting usages in base further supports this. 

There is a suggestion in this thread worth keeping, though - supporting unicode 
\in would be a nice addition for those of us who embrace supersets of ascii.

// Tomas 

Re: [julia-users] Re: OT: make githup notifications un-read?

[Tomas Lycken]

The Gmail app on Android can handle several Google accounts (at least on my 
phone :P) so the race might not be lost yet :)

// T

On Wednesday, October 28, 2015 at 11:09:54 AM UTC+1, Andreas Lobinger wrote:
>
> Hello colleagues,
>
> thanks for the answers. I thought i'm missing some github configuration, 
> but i see the some third party interaction is needed. As i use a different 
> account for my android phone working with this gmail account can't be 
> enabled.
>
>
>

Re: [julia-users] Re: Avoid type Unions in fields?

[Tomas Lycken]

Yeah, that's a good point. When you give the compiler the assurance that 
"I'm not going to change the type of x here", you also loose the 
possibility to do that :) Type instabilities are a major performance 
bottleneck (one of the most common performance problems, actually...) but 
in many applications simplicity is more important than performance, and 
then your type union is just fine.

// T

On Wednesday, October 28, 2015 at 1:10:41 PM UTC+1, Stefan Karpinski wrote:
>
> The key behavioral difference between these two is that with the 
> non-parametric union version you can change both the value and the type of 
> the field of a MyType object after construction, whereas in the parametric 
> version you can change the value of the field after an object has been 
> constructed, but not the type – since the object is either a MyType{Int} or 
> a MyType{Float64}.
>
> On Wed, Oct 28, 2015 at 3:21 AM, Gnimuc Key <gnim...@gmail.com 
> > wrote:
>
>> that makes sense. many thanks!
>>
>> 在 2015年10月28日星期三 UTC+8下午3:14:26，Tomas Lycken写道：
>>
>>> With a type declared like that, any access of the field x will be type 
>>> unstable, which means that Julia’s JIT compiler will emit much more 
>>> defensive, and thus slower, code.
>>>
>>> The solution is to declare a *parametric* type:
>>>
>>> type MyType{T<:Union{Int64, Float64}}
>>> x::T
>>> end
>>>
>>> That way, MyType(3) and MyType(3.0) will now be instances of different 
>>> types, and the compiler can generate fast code. (You can still write 
>>> functions that dispatch on just MyType, so you don’t loose any 
>>> expressive power…)
>>>
>>> // T
>>>
>>> On Wednesday, October 28, 2015 at 5:10:09 AM UTC+1, Gnimuc Key wrote:
>>>
>>> Avoid type Unions in fields ¶
>>>>>
>>>>> <http://docs.julialang.org/en/latest/manual/style-guide/#avoid-strange-type-unions>
>>>>> type MyType
>>>>> ...
>>>>> x::Union{Void,T}
>>>>> end
>>>>
>>>>  
>>>> I'm wondering whether it's a good style or not to write something like 
>>>> this:
>>>>
>>>> type MyType
>>>> ...
>>>> x::Union{Int64,Float64}end
>>>>
>>>> what's the side effects of using Union like this?
>>>>
>>> ​
>>>
>>
>

[julia-users] Re: Avoid type Unions in fields?

[Tomas Lycken]

With a type declared like that, any access of the field x will be type 
unstable, which means that Julia’s JIT compiler will emit much more 
defensive, and thus slower, code.

The solution is to declare a *parametric* type:

type MyType{T<:Union{Int64, Float64}}
x::T
end

That way, MyType(3) and MyType(3.0) will now be instances of different 
types, and the compiler can generate fast code. (You can still write 
functions that dispatch on just MyType, so you don’t loose any expressive 
power…)

// T

On Wednesday, October 28, 2015 at 5:10:09 AM UTC+1, Gnimuc Key wrote:

Avoid type Unions in fields ¶
>>
>> 
>> type MyType
>> ...
>> x::Union{Void,T}
>> end
>
>  
> I'm wondering whether it's a good style or not to write something like 
> this:
>
> type MyType
> ...
> x::Union{Int64,Float64}end
>
> what's the side effects of using Union like this?
>
​

[julia-users] Re: alternate LISP integration ...

[Tomas Lycken]

Have you tried julia --lisp from the command line? Works on my machine…

// T

On Monday, October 26, 2015 at 10:36:05 PM UTC+1, cdm wrote:


> somewhat related ...
>
> is it still possible to get
> the femtolisp REPL in the
> post v0.4.0 world ... ?
>
> thanks,
>
> cdm
>
​

[julia-users] Re: Pkg.add("DecisionTree") getting error, failed pocess: Process(`git ' --worktree = C:\Users.....

[Tomas Lycken]

Is it possible for you to scroll right in the first error message, to show 
the entire message? Easiest is probably if you could copy the *text* of the 
message, and paste it here.

// T

On Monday, October 26, 2015 at 8:26:07 PM UTC+1, aar…@udel.edu wrote:

Hi I am running my Julia code in Juno/LT. I am trying to add the decision 
> tree package but I am getting a major error seen in the screenshot below. 
> Any help would be much appreciated.
>
>
>
> 
>
​

Re: [julia-users] A question of Style: Iterators into regular Arrays

[Tomas Lycken]

Better yet, since we already have both AbstractVector, AbstractMatrix, 
AbstractArray, AbstractString, AbstractFloat and a couple of others (try 
typing Abstract in the REPL…) it might be time to rename Integer to 
AbstractInteger. I have a hard time thinking the confusion between Int and 
Integer would be reduced just because we also had Arr and Array et al - 
rather, we’d have *several* pairs of types where it isn’t entirely obvious 
that one is abstract and one is concrete.

Renaming Integer to AbstractInteger would probably cause massive breakage, 
though, so it’d have to be done with care. The difference between Int, Int32
/Int64 and Integer is well documented (see e.g. here 

 
and here 
), 
but it seems to me that people stumble on this often enough that a naming 
change might be well motivated anyway.
// T

On Monday, October 26, 2015 at 8:26:08 PM UTC+1, Scott Jones wrote:

>One possible naming scheme could be to follow the example of Int and 
> Integer and have Vec, Mat, Arr be the concrete types and Vector, Matrix and 
> Array be the abstract types. I'm really not sure this would be worth the 
> trouble at this point or if we really want the AbstractArray names to be 
> any shorter.
>
> That sounds like a quite good idea, which, if carried out completely, 
> could eliminate some inconsistencies in the naming of abstract vs. concrete 
> types, that have been causing people grief.
>
> So:
>
>> Abstract Concrete
>> Signed (Integer) Int*
>> Unsigned (Integer) UInt*
>> Float Flt*
>> Decimal Dec*
>> Array Arr
>> Vector Vec
>> Matrix Mat
>> String Str (maybe Str{Binary}, Str{ASCII}, Str{UTF8}, Str{UTF16}, 
>> Str{UTF32})
>
>
> ​

[julia-users] Re: Julia and Object-Oriented Programming

[Tomas Lycken]

I’ve never seen a problem that foo.bar(x,y) can solve, but bar(foo, x, y) 
can’t.

There *are* things, however, that are pretty easy to do in OO languages 
that can’t be done in Julia; one such thing is adding/changing behavior 
through inheritance. That’s pretty easy to work around, though - instead of 
the following (which won’t work in Julia, but using Julia syntax to make 
comparison easier)

immutable Foo
x
end

foo(f::Foo) = f.x)
bar(f::Foo) = 2 * f.x

immutable Bar <: Foo
y
end

bar(b::Bar) = b.y * b.x

where Bar has an x field inherited from Foo, and overrides behavior for the 
bar function, you can easily do this:

abstract AbstractFoo
immutable Foo
x
end

foo(f::Foo) = f.x
bar(f::Foo) = 2 * f.x

immutable Bar
y
f::Foo
end

foo(b::Bar) = foo(b.f)
bar(b::Bar) = b.y * b.f.x

In other words, by using *composition* instead of inheritance, you can 
extend and amend the behavior of any object with all the power you have in 
inheritance. (You could also create an abstract type that both Foo and Bar 
inherits, to make them part of the same type tree and make dispatch a 
little simpler.) This is heavily relied on in a couple of recent additions 
to Interpolations.jl, that provide axis scaling and extrapolation through 
composition - had I written that code in Java or C# (or C++), I probably 
would have used inheritance instead.

But I agree with Tobias and Simon - there is very little reason to try to 
make Julia more object-oriented. All the power you get from OO is still 
there, but you might need to think a little differently to harness it. If 
you have a concrete problem, along the lines of “this is how I’d do it in 
OO, but I don’t know how to do it in Julia”, the community is usually 
pretty good att figuring something out. A recent example was that someone 
wanted man.eat(food) rather than eat(man, food). A little thinking outside 
the box turned out the IMO superior solution - instead of changing the 
language to make it read better, just change the verb: feed(man, food).

Julia isn’t object-oriented, and will probably never be. But that doesn’t 
make it less powerful :)

// T

On Sunday, October 18, 2015 at 8:32:06 PM UTC+2, Tobias Knopp wrote:

Julia is fully object oriented. The core of OO is not to write a dot 
> between an object and an associated function but that one has virtual 
> functions. Multiple dispatch provides just that.
>
> Cheers
>
> Tobi
>
> Am Sonntag, 18. Oktober 2015 15:15:50 UTC+2 schrieb Simon Danisch:
>>
>> This design has at least 3 issues, since you can't have instances of a 
>> module.
>> In general:
>> I personally would value the discussion of why Julia needs more OOP 
>> constructs much more, If one can show that there are terrible flaws in 
>> Julia's model which are swiftly solvable with OOP.
>> My guess is, that the argument "I learned X and can't transfer it to 
>> Julia, so Julia needs to change", can't really convince anyone to start 
>> implementing something rather involved.
>>
>> There are a lot of comments on people trying to bring OOP to Julia, which 
>> are best summarized by: give Julia's model a chance and don't stick to OOP 
>> - it will pay off.
>> And this kind of comment is pretty understandable in my opinion, since 
>> Julia is quite a sophistaced language with plenty of constructs to avoid 
>> OOP.
>> I don't see much value in OOP and Julia's multiple dispatch is actually 
>> one of the things that brought me here ( I exclusively worked with OOP 
>> languages before).
>> Besides the current flaws, I think it's a better model for numerical 
>> computing, it yields concise code, easier maintenance and it's easier to 
>> add feature to an existing code base than it is in OOP.
>>
>> I think Julia should move forward by not mimicking OOP further, but 
>> instead improve the current model - there's lots to do already.
>> Wasting time by chasing after features from other languages will make it 
>> much harder to turn Julia into a well rounded, sound language. (this only 
>> applies to feature that get chased without a concrete problem)
>>
>> Best,
>> Simon
>>
>>
>> Am Sonntag, 18. Oktober 2015 14:41:58 UTC+2 schrieb Sisyphuss:
>>>
>>> When I'm learning Julia, I am always thinking what is the correct way to 
>>> do OOP in this language. It seems to me that what I learned in C++ does not 
>>> apply in Julia.
>>>
>>> It took me long to realize that the equivalent of Class of C++ in Julia 
>>> is not Type, but Module. Module is the basic function unit in Julia.
>>>
>>> Thus, a Class in Julia is like
>>> module ClassName # class Name {
>>> using# include<> // should be outside
>>> import   # include<>
>>> export  function # public  function;
>>> var = 1  # private static var;
>>> end  # }
>>> This provides the same structure as C++.
>>>
>>> However, this design has two issues:
>>> 1) The visit control is not as fine-grained as 

Re: [julia-users] Re: Everything I wrote in version .3 is now 'depreciated'

[Tomas Lycken]

… julia doesn’t like ‘foo ()’ anymore, she now prefers ‘foo()’ …

In a recent thread I vowed to start helping with enforcing the Julia 
Community Standards , so I guess 
it’s now time to walk the walk. In the Standards, we can read

While “Julia” is a female name in many parts of the world, the programming 
language is not a person and does not have a gender.

In other words, don’t say “she” about the Julia language; say “it”.

On a related note, I’m not sure I’ve seen your name on the list before, 
Forrest - welcome to the community! :)

// T

On Saturday, October 17, 2015 at 5:26:39 PM UTC+2, Forrest Curo wrote:

Thanks, all!
>
> Most of my booboos seem to be due to the fact that julia doesn't like 
> 'foo ()' 
> anymore, she now prefers 'foo()' which I suppose makes the parsing simpler.
> And I now know where to find that pesky tk button code...
>
> On Sat, Oct 17, 2015 at 8:00 AM, Kristoffer Carlsson  > wrote:
>
>> You can start julia with -depwarn=no to turn off deprecation warnings.
>>
>> On a personal note, I think that the time between type deprecation and 
>> the release was possibly a bit short. It feels that whatever package I 
>> import now I get pages of deprecation warnings. This is particularly 
>> inconvenient in IJulia sessions.
>
>
> ​

[julia-users] Re: Windows: enforce update of ENV["PATH"] after modifying it.

[Tomas Lycken]

Modifying environment variables on Windows usually requires at least a sign 
out-sign in again. Can you try that?

// T

On Friday, October 16, 2015 at 2:06:26 PM UTC+2, bernhard wrote:
>
> Hi 
> can anyone help me with this? When PATH is modified this is not 
> immediately reflected in Julia (even after closing and starting Julia 
> again). I wonder if this can be done without a reboot.
> I am not sure whether this is at all a Julia issue though.
>
> Bernhard
>