Re: [julia-users] Re: ANN: Kip.jl an alternative module system

[Stefan Karpinski]

There's also https://github.com/Rory-Finnegan/Playground.jl. I suspect that
a large part of the issue with upper bounds in REQUIRE is that they aren't
automatically discovered.

On Mon, Dec 21, 2015 at 10:59 PM, Tony Kelman  wrote:

> The current REQUIRE specification allows for upper bounds to deal with
> this.
>
> The only issue there is an upper bound by any reverse-dep package holds
> the package back for all other uses. Using a per-project separate sandbox
> might help with that, as in virtualenv, which we don't have a single
> standard way of doing just yet. Setting JULIA_PKGDIR works as a decent
> approximation.

Re: [julia-users] Re: ANN: Kip.jl an alternative module system

[Tony Kelman]

The current REQUIRE specification allows for upper bounds to deal with this.

The only issue there is an upper bound by any reverse-dep package holds the 
package back for all other uses. Using a per-project separate sandbox might 
help with that, as in virtualenv, which we don't have a single standard way of 
doing just yet. Setting JULIA_PKGDIR works as a decent approximation.

Re: [julia-users] Re: ANN: Kip.jl an alternative module system

[Stefan Karpinski]

The share library issue could be dealt with by separating library
installation in loading into a lower layer from pure Julia packages, and
requiring all packages to agree on a version of each library that they can
share. This is less of an issue since system libraries tend to be pretty
stable.

On Mon, Dec 21, 2015 at 10:07 PM, Jake Rosoman  wrote:

> Yea though in that case since it's intentional duplication you would be
> able to change the name of the package.
>
> My issue with the current system is that any package that gets used a lot
> develops a responsibility to not change since all packages that depend on
> it would be broken until everyone finds the time to update their own
> packages. Repeat for that packages dependencies. Being able to run two
> versions gives people a grace period to update where everything still
> works, be it with more code generated. Or at least thats the dream, I need
> to figure out exactly how well it will actually work for Julia considering
> the issues Stefan mentioned.
>
> Another solution to this problem could be to append a version number to
> the name of your package when you make you first significant change.
>
>> On Dec 22 2015, at 2:43 pm, Daniel Carrera  wrote:
>> I think there might be some merit in having two versions of a package
>> installed (e.g. "experimental vs stable"), but I certainly wouldn't want to
>> RUN two versions of the same package at the same time.
>>
>> Cheers,
>> Daniel.
>>
>> On Sunday, 20 December 2015 17:23:01 UTC+1, Stefan Karpinski wrote:
>>
>> This seems to be a design based largely on npm.
>>
>> The idea of modules (and packages) being unnamed internally is appealing,
>> largely for the prospect of it being possible to rename packages without
>> breaking code that uses them although I'm not certain this is realistic.
>>
>> I'm not sold on the idea that it makes sense to load multiple versions of
>> packages. There are some major issues:
>>
>> *Generic functions.* What happens when two different versions of a
>> package both introduce separate generic function that other packages are
>> supposed to share? One of the most confusing situations I see people
>> encounter is when they've reloaded some code that defines a type and have a
>> generic function that dispatches the old version of that type and they try
>> to call the function on the new version of the type. The dispatch doesn't
>> work, of course, but it looks like it should. Very confusing. This seems to
>> be intentionally introducing a far worse form of that. It's possible that a
>> coherent story can be told about how to deal with this, but without that
>> story this seems pretty hard to swallow.
>>
>> *Shared system libraries.* What do you do when two different versions of
>> a package want to load different versions of system libraries? In theory
>> dlopen can do this but in practice it does not work well at all. JavaScript
>> doesn't have to deal with this so npm can just ignore the issue.
>>
>> *Code generation.* Julia is already a bit of a beast when it comes to
>> the amount of code that is generated. This makes that N times worse, for
>> some unknown and potentially fairly large N.
>>
>> On Sun, Dec 20, 2015 at 10:05 AM, Tom Breloff  wrote:
>>
>> My reaction is the same as Tim Holy's.  While I agree there are aspects
>> of Julia's modules which are not perfect, it's not clear at all how your
>> package changes the workflow.  In fact, I don't understand anything about
>> your package.  Could you please try to write up the design thoughts behind
>> what you are doing, so that we can understand the high level concepts that
>> you are attempting?
>>
>> On Sun, Dec 20, 2015 at 6:24 AM, Tim Holy  wrote:
>>
>> After reading your README example, I'm still left wondering how one works
>> with
>> Kip, or how it fixes the problems you're describing. To me it's not at all
>> obvious how your example "illustrates" the statements you make in the
>> prose.
>> You might consider explaining the meaning of the various arguments to
>> @require, what an "index" file is and what its format should be, and
>> exactly
>> what the call to the emit function is supposed to demonstrate.
>>
>> Best,
>> --Tim
>>
>> On Saturday, December 19, 2015 11:26:53 PM Jake Rosoman wrote:
>> > I forgot to actually link to the project <
>> https://github.com/jkroso/Kip.jl>
>> >
>> > On Sunday, December 20, 2015 at 8:25:04 PM UTC+13, Jake Rosoman wrote:
>> > > Julia's module system is the one part of it I feel confident enough
>> to say
>> > > is bad. It can't handle several versions of the same package. Is hard
>> (or
>> > > impossible?) to depend on packages that aren't in the registry and
>> hard to
>> > > add (controversial) things to the registry. I also find it ugly and
>> hard
>> > > to
>> > > use but now I'm getting into opinions so I'll stop.
>> > >
>> > > Kip solves all these problems and works fine alongside Julia's current
>> > > module system so you can try it out now. I hope that eventually we can

Re: [julia-users] Re: ANN: Kip.jl an alternative module system

[Jake Rosoman]

Yea though in that case since it's intentional duplication you would be able
to change the name of the package.

  

My issue with the current system is that any package that gets used a lot
develops a responsibility to not change since all packages that depend on it
would be broken until everyone finds the time to update their own packages.
Repeat for that packages dependencies. Being able to run two versions gives
people a grace period to update where everything still works, be it with more
code generated. Or at least thats the dream, I need to figure out exactly how
well it will actually work for Julia considering the issues Stefan mentioned.

  

Another solution to this problem could be to append a version number to the
name of your package when you make you first significant change.

> On Dec 22 2015, at 2:43 pm, Daniel Carrera  wrote:  

>

> I think there might be some merit in having two versions of a package
installed (e.g. "experimental vs stable"), but I certainly wouldn't want to
RUN two versions of the same package at the same time.

>

>  

>

> Cheers,

>

> Daniel.  
  
On Sunday, 20 December 2015 17:23:01 UTC+1, Stefan Karpinski wrote:

>

>> This seems to be a design based largely on npm.

>>

>>  

>>

>> The idea of modules (and packages) being unnamed internally is appealing,
largely for the prospect of it being possible to rename packages without
breaking code that uses them although I'm not certain this is realistic.

>>

>>  

>>

>> I'm not sold on the idea that it makes sense to load multiple versions of
packages. There are some major issues:

>>

>>  

>>

>> **Generic functions.** What happens when two different versions of a
package both introduce separate generic function that other packages are
supposed to share? One of the most confusing situations I see people encounter
is when they've reloaded some code that defines a type and have a generic
function that dispatches the old version of that type and they try to call the
function on the new version of the type. The dispatch doesn't work, of course,
but it looks like it should. Very confusing. This seems to be intentionally
introducing a far worse form of that. It's possible that a coherent story can
be told about how to deal with this, but without that story this seems pretty
hard to swallow.  

>>

>>  

>>

>> **Shared system libraries.** What do you do when two different versions of
a package want to load different versions of system libraries? In theory
dlopen can do this but in practice it does not work well at all. JavaScript
doesn't have to deal with this so npm can just ignore the issue.  
  

>>

>> **Code generation.** Julia is already a bit of a beast when it comes to the
amount of code that is generated. This makes that N times worse, for some
unknown and potentially fairly large N.

>>

>>  

>>

>> On Sun, Dec 20, 2015 at 10:05 AM, Tom Breloff 
wrote:  

>>

>>> My reaction is the same as Tim Holy's.  While I agree there are aspects of
Julia's modules which are not perfect, it's not clear at all how your package
changes the workflow.  In fact, I don't understand anything about your
package.  Could you please try to write up the design thoughts behind what you
are doing, so that we can understand the high level concepts that you are
attempting?

>>>

>>>  

>>>

>>> On Sun, Dec 20, 2015 at 6:24 AM, Tim Holy  wrote:  

>>>

 After reading your README example, I'm still left wondering how one works
with  
Kip, or how it fixes the problems you're describing. To me it's not at all  
obvious how your example "illustrates" the statements you make in the prose.  
You might consider explaining the meaning of the various arguments to  
@require, what an "index" file is and what its format should be, and exactly  
what the call to the emit function is supposed to demonstrate.  
  
Best,  
\--Tim  
  
On Saturday, December 19, 2015 11:26:53 PM Jake Rosoman wrote:  
> I forgot to actually link to the project
<[https://github.com/jkroso/Kip.jl](https://github.com/jkroso/Kip.jl)>  
>  
> On Sunday, December 20, 2015 at 8:25:04 PM UTC+13, Jake Rosoman wrote:  
> > Julia's module system is the one part of it I feel confident enough
to say  
> > is bad. It can't handle several versions of the same package. Is
hard (or  
> > impossible?) to depend on packages that aren't in the registry and
hard to  
> > add (controversial) things to the registry. I also find it ugly and
hard  
> > to  
> > use but now I'm getting into opinions so I'll stop.  
> >  
> > Kip solves all these problems and works fine alongside Julia's
current  
> > module system so you can try it out now. I hope that eventually we
can  
> > replace Julia's module system if people generally agree that it's
worth  
> > doing. I've created a poll to measure the communities opinion  
> > <> and you can change your vote at any
time so feel  



 > > free to say no now but follow 

Re: [julia-users] Why is the order of these loops barely affects the time?

[elextr]

N is a mutable global, that means that any piece of code anywhere can 
change it, so every use has to test its type, which takes time.  Making it 
const means it can't change, so its type can't change.

Cheers
Lex

On Tuesday, 22 December 2015 09:35:30 UTC+10, Joaquim Masset Lacombe Dias 
Garcia wrote:
>
> Const made the big difference, 
> But what is the problem with the inferred type? 
>
> Joaquim 
>
> > On 21 de dez de 2015, at 20:30, Milan Bouchet-Valat  > wrote: 
> > 
> > Le lundi 21 décembre 2015 à 14:04 -0800, Joaquim Masset Lacombe Dias 
> > Garcia a écrit : 
> >> Why is the order of these loops barely affects the time? 
> > Before doing this kind of comparison, you should ensure your code is 
> > written to run efficiently. For that, you need to do 
> > const N = 100 
> > 
> > Else, the type of the variables isn't correctly inferred. Also, you 
> > need to call the functions once to exclude compilation time from the 
> > measurement. 
> > 
> > After changing this (and fixing the name of the second argument for 
> > cba), I get the maximum time difference between abc and cba, the latter 
> > being 3x faster than the former (4x after adding @inbounds). 
> > 
> > 
> > Regards 
> > 
> > 
> >> N=100 
> >> x=rand(N,N,N) 
> >> y=rand(N,N,N) 
> >> 
> >> function abc(x::Array{Float64,3},y::Array{Float64,3}) 
> >>for a=1:N 
> >>for b=1:N 
> >>for c=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> function acb(x::Array{Float64,3},y::Array{Float64,3}) 
> >>for a=1:N 
> >>for c=1:N 
> >>for b=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> function bac(x::Array{Float64,3},y::Array{Float64,3}) 
> >>for b=1:N 
> >>for a=1:N 
> >>for c=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> function bca(x::Array{Float64,3},y::Array{Float64,3}) 
> >>for b=1:N 
> >>for c=1:N 
> >>for a=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> function cab(x::Array{Float64,3},y::Array{Float64,3}) 
> >>for c=1:N 
> >>for a=1:N 
> >>for b=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> function cba(x::Array{Float64,3},xx::Array{Float64,3}) 
> >>for c=1:N 
> >>for b=1:N 
> >>for a=1:N 
> >>y[a,b,c]=x[a,b,c] 
> >>end 
> >>end 
> >>end 
> >> end 
> >> 
> >> @time abc(x,y) 
> >> @time acb(x,y) 
> >> @time bac(x,y) 
> >> @time bca(x,y) 
> >> @time cab(x,y) 
> >> @time cba(x,y) 
>

Re: [julia-users] Re: ANN: Kip.jl an alternative module system

[Daniel Carrera]

I think there might be some merit in having two versions of a package 
installed (e.g. "experimental vs stable"), but I certainly wouldn't want to 
RUN two versions of the same package at the same time.

Cheers,
Daniel.

On Sunday, 20 December 2015 17:23:01 UTC+1, Stefan Karpinski wrote:
>
> This seems to be a design based largely on npm.
>
> The idea of modules (and packages) being unnamed internally is appealing, 
> largely for the prospect of it being possible to rename packages without 
> breaking code that uses them although I'm not certain this is realistic.
>
> I'm not sold on the idea that it makes sense to load multiple versions of 
> packages. There are some major issues:
>
> *Generic functions.* What happens when two different versions of a 
> package both introduce separate generic function that other packages are 
> supposed to share? One of the most confusing situations I see people 
> encounter is when they've reloaded some code that defines a type and have a 
> generic function that dispatches the old version of that type and they try 
> to call the function on the new version of the type. The dispatch doesn't 
> work, of course, but it looks like it should. Very confusing. This seems to 
> be intentionally introducing a far worse form of that. It's possible that a 
> coherent story can be told about how to deal with this, but without that 
> story this seems pretty hard to swallow.
>
> *Shared system libraries.* What do you do when two different versions of 
> a package want to load different versions of system libraries? In theory 
> dlopen can do this but in practice it does not work well at all. JavaScript 
> doesn't have to deal with this so npm can just ignore the issue.
>
> *Code generation.* Julia is already a bit of a beast when it comes to the 
> amount of code that is generated. This makes that N times worse, for some 
> unknown and potentially fairly large N.
>
> On Sun, Dec 20, 2015 at 10:05 AM, Tom Breloff  > wrote:
>
>> My reaction is the same as Tim Holy's.  While I agree there are aspects 
>> of Julia's modules which are not perfect, it's not clear at all how your 
>> package changes the workflow.  In fact, I don't understand anything about 
>> your package.  Could you please try to write up the design thoughts behind 
>> what you are doing, so that we can understand the high level concepts that 
>> you are attempting?
>>
>> On Sun, Dec 20, 2015 at 6:24 AM, Tim Holy > > wrote:
>>
>>> After reading your README example, I'm still left wondering how one 
>>> works with
>>> Kip, or how it fixes the problems you're describing. To me it's not at 
>>> all
>>> obvious how your example "illustrates" the statements you make in the 
>>> prose.
>>> You might consider explaining the meaning of the various arguments to
>>> @require, what an "index" file is and what its format should be, and 
>>> exactly
>>> what the call to the emit function is supposed to demonstrate.
>>>
>>> Best,
>>> --Tim
>>>
>>> On Saturday, December 19, 2015 11:26:53 PM Jake Rosoman wrote:
>>> > I forgot to actually link to the project <
>>> https://github.com/jkroso/Kip.jl>
>>> >
>>> > On Sunday, December 20, 2015 at 8:25:04 PM UTC+13, Jake Rosoman wrote:
>>> > > Julia's module system is the one part of it I feel confident enough 
>>> to say
>>> > > is bad. It can't handle several versions of the same package. Is 
>>> hard (or
>>> > > impossible?) to depend on packages that aren't in the registry and 
>>> hard to
>>> > > add (controversial) things to the registry. I also find it ugly and 
>>> hard
>>> > > to
>>> > > use but now I'm getting into opinions so I'll stop.
>>> > >
>>> > > Kip solves all these problems and works fine alongside Julia's 
>>> current
>>> > > module system so you can try it out now. I hope that eventually we 
>>> can
>>> > > replace Julia's module system if people generally agree that it's 
>>> worth
>>> > > doing. I've created a poll to measure the communities opinion
>>> > >  and you can change your vote at any time so 
>>> feel
>>> > > free to say no now but follow the discussion.
>>>
>>>
>>
>

Re: [julia-users] Why is the order of these loops barely affects the time?

[Joaquim Dias Garcia]

Const made the big difference,
But what is the problem with the inferred type?

Joaquim

> On 21 de dez de 2015, at 20:30, Milan Bouchet-Valat  wrote:
> 
> Le lundi 21 décembre 2015 à 14:04 -0800, Joaquim Masset Lacombe Dias
> Garcia a écrit :
>> Why is the order of these loops barely affects the time?
> Before doing this kind of comparison, you should ensure your code is
> written to run efficiently. For that, you need to do
> const N = 100
> 
> Else, the type of the variables isn't correctly inferred. Also, you
> need to call the functions once to exclude compilation time from the
> measurement.
> 
> After changing this (and fixing the name of the second argument for
> cba), I get the maximum time difference between abc and cba, the latter
> being 3x faster than the former (4x after adding @inbounds).
> 
> 
> Regards
> 
> 
>> N=100
>> x=rand(N,N,N)
>> y=rand(N,N,N)
>> 
>> function abc(x::Array{Float64,3},y::Array{Float64,3})
>>for a=1:N
>>for b=1:N
>>for c=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> function acb(x::Array{Float64,3},y::Array{Float64,3})
>>for a=1:N
>>for c=1:N
>>for b=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> function bac(x::Array{Float64,3},y::Array{Float64,3})
>>for b=1:N
>>for a=1:N
>>for c=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> function bca(x::Array{Float64,3},y::Array{Float64,3})
>>for b=1:N
>>for c=1:N
>>for a=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> function cab(x::Array{Float64,3},y::Array{Float64,3})
>>for c=1:N
>>for a=1:N
>>for b=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> function cba(x::Array{Float64,3},xx::Array{Float64,3})
>>for c=1:N
>>for b=1:N
>>for a=1:N
>>y[a,b,c]=x[a,b,c]
>>end
>>end
>>end
>> end
>> 
>> @time abc(x,y)
>> @time acb(x,y)
>> @time bac(x,y)
>> @time bca(x,y)
>> @time cab(x,y)
>> @time cba(x,y)

Re: [julia-users] Why is the order of these loops barely affects the time?

[Milan Bouchet-Valat]

Le lundi 21 décembre 2015 à 14:04 -0800, Joaquim Masset Lacombe Dias
Garcia a écrit :
> Why is the order of these loops barely affects the time?
Before doing this kind of comparison, you should ensure your code is
written to run efficiently. For that, you need to do
const N = 100

Else, the type of the variables isn't correctly inferred. Also, you
need to call the functions once to exclude compilation time from the
measurement.

After changing this (and fixing the name of the second argument for
cba), I get the maximum time difference between abc and cba, the latter
being 3x faster than the former (4x after adding @inbounds).


Regards


> N=100
> x=rand(N,N,N)
> y=rand(N,N,N)
> 
> function abc(x::Array{Float64,3},y::Array{Float64,3})
> for a=1:N
> for b=1:N
> for c=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> function acb(x::Array{Float64,3},y::Array{Float64,3})
> for a=1:N
> for c=1:N
> for b=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> function bac(x::Array{Float64,3},y::Array{Float64,3})
> for b=1:N
> for a=1:N
> for c=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> function bca(x::Array{Float64,3},y::Array{Float64,3})
> for b=1:N
> for c=1:N
> for a=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> function cab(x::Array{Float64,3},y::Array{Float64,3})
> for c=1:N
> for a=1:N
> for b=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> function cba(x::Array{Float64,3},xx::Array{Float64,3})
> for c=1:N
> for b=1:N
> for a=1:N
> y[a,b,c]=x[a,b,c]
> end
> end
> end
> end
> 
> @time abc(x,y)
> @time acb(x,y)
> @time bac(x,y)
> @time bca(x,y)
> @time cab(x,y)
> @time cba(x,y)

[julia-users] Re: Why is the order of these loops barely affects the time?

[Andre Bieler]

Hi Joaquim,

First, you need to call the functions once before timing them,
otherwise the compilation time is also included.

Second, N is a global variable in your example, which is slow.
If you want to use N as a global variable, make it a constant with
const N = 100
this will increase the speed of all functions and also the difference 
between the various implementations.
Otherwise you can put inside every function something like
N = size(x,1)

Then your last function definition has an argument xx, instead of y.

And finally, you actually do not need to specify all the function 
parameters the way you did in order to get speed.
The following will also do:

const N=100

x=rand(N,N,N)
y=rand(N,N,N)

function abc(x,y)
for a=1:N
for b=1:N
for c=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function acb(x,y)
for a=1:N
for c=1:N
for b=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function bac(x,y)
for b=1:N
for a=1:N
for c=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function bca(x,y)
for b=1:N
for c=1:N
for a=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function cab(x,y)
for c=1:N
for a=1:N
for b=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function cba(x,y)
for c=1:N
for b=1:N
for a=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end

@time abc(x,y)
@time acb(x,y)
@time bac(x,y)
@time bca(x,y)
@time cab(x,y)
@time cba(x,y)
println()
@time abc(x,y)
@time acb(x,y)
@time bac(x,y)
@time bca(x,y)
@time cab(x,y)
@time cba(x,y)



cheers,
andre

[julia-users] Why is the order of these loops barely affects the time?

[Joaquim Masset Lacombe Dias Garcia]

Why is the order of these loops barely affects the time?

N=100
x=rand(N,N,N)
y=rand(N,N,N)

function abc(x::Array{Float64,3},y::Array{Float64,3})
for a=1:N
for b=1:N
for c=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function acb(x::Array{Float64,3},y::Array{Float64,3})
for a=1:N
for c=1:N
for b=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function bac(x::Array{Float64,3},y::Array{Float64,3})
for b=1:N
for a=1:N
for c=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function bca(x::Array{Float64,3},y::Array{Float64,3})
for b=1:N
for c=1:N
for a=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function cab(x::Array{Float64,3},y::Array{Float64,3})
for c=1:N
for a=1:N
for b=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end
function cba(x::Array{Float64,3},xx::Array{Float64,3})
for c=1:N
for b=1:N
for a=1:N
y[a,b,c]=x[a,b,c]
end
end
end
end

@time abc(x,y)
@time acb(x,y)
@time bac(x,y)
@time bca(x,y)
@time cab(x,y)
@time cba(x,y)

[julia-users] Re: redirect error output from julia repl to text file ?

[jda]

Indeed, my oversight thank you!

[julia-users] Re: redirect error output from julia repl to text file ?

[elextr]

Example from the manual:


   - run(pipeline(`dothings`, stdout="out.txt", stderr="errs.txt"))

Cheers
Lex

On Tuesday, 22 December 2015 06:41:34 UTC+10, jda wrote:
>
> I tried the command 
>
> path = pathtoscript.jl;
> output = pathtologfile.txt;
> run(pipeline(` julia $path` ,output))
>
> but it does not catch error messages, only standard output.  I have done a 
> number of searches around for a solution but what I find is either over my 
> head or outdated. Would greatly appreciate any help on a basic solution! 
>  Thank you in advance !
>

[julia-users] redirect error output from julia repl to text file ?

[jda]

I tried the command 

path = pathtoscript.jl;
output = pathtologfile.txt;
run(pipeline(` julia $path` ,output))

but it does not catch error messages, only standard output.  I have done a 
number of searches around for a solution but what I find is either over my 
head or outdated. Would greatly appreciate any help on a basic solution! 
 Thank you in advance !

Re: [julia-users] Re: Survey: what documentation platforms do you use? Are you happy?

[Stefan Karpinski]

We're slowly but surely headed in that direction.

On Mon, Dec 21, 2015 at 12:38 PM,  wrote:

> In the future, perhaps all Julia documentation, both for Base and for
> registered packages, will be in a central location, indexed and
> cross-linked (like Matlab, Mathematica, SciPy, only even better...) Perhaps
> written in a single documentation system written in Julia. I'm looking
> forward to helping out.
>

[julia-users] Re: Array view without a single element

[Tomas Lycken]

The algorithm I want to implement is fairly simple - it's a basic change 
making problem: "given the following coins [20 coins with integer-values 
between 1 and 50] how many ways is there to combine them to get a total 
value of 150? two coins with equal value contribute one time each to a 
combination involving only one of them."

My approach was based on this SO answer 
, basically figuring it out 
based on how many ways there are to get a total value of 150-x from the 
list of all coins except x. I might be able to remove and re-insert the 
value, but ordering is important (since I'll loop over the values, and I 
only want to test each value once on each level).

I'll have to think about this some more, but at least now I know that my 
naïve approach is impossible to optimize regardless of my inability to 
construct the indexing array efficiently :)

Thanks!

// T

On Monday, December 21, 2015 at 6:57:39 PM UTC+1, Tero Frondelius wrote:
>
> Hi
> Can you use a macro to produce your twenty nested loops? 

[julia-users] Re: Different result in pmap vs remotecall

[lmagarshack]

I should probably just call *searchsortedfirst* in my *pmap* anyways and 
*deref* everything in a "reduce" stage...

But I just thought it was an intriguing issue : )

[julia-users] Re: Segmentation fault after deep-copying SharedArray

[Daniel Arndt]

That seems to be it. I'll prepare an issue and pull request. Thanks for the 
rubber duck debugging!

On Monday, 21 December 2015 14:44:58 UTC-4, Daniel Arndt wrote:
>
> The type of the array being returned seems to be determined by the 
> "similar" function:
>
> julia> sa = SharedArray(Float64, 5, init = S -> S[Base.localindexes(S)] = 
> 0)
> 5-element SharedArray{Float64,1}:
>  0.0
>  0.0
>  0.0
>  0.0
>  0.0
>
>
> julia> typeof(similar(sa))
> Array{Float64,1}
>
> Perhaps that is the underlying issue.
>
> On Monday, 21 December 2015 14:00:46 UTC-4, Daniel Arndt wrote:
>>
>> I've been trying to isolate a defect I have in my code for the past 
>> couple days, and it seems like SharedArrays have troubles with deepcopying.
>>
>> Looking at the source, this should be supported (Julia 0.4.2):
>>
>> https://github.com/JuliaLang/julia/blob/v0.4.2/base/sharedarray.jl#L201
>>
>> I've created a small stripped down version of the problem I am 
>> encountering.
>>
>> type A
>> sa::SharedArray{Float64,1}
>> end
>>
>>
>> function A(n::Int)
>> sa = SharedArray(Float64, n, init = S -> S[Base.localindexes(S)] = 0)
>> return A(sa)
>> end
>>
>> function Base.show(io::IO, a::A)
>> print(io,a.sa)
>> end
>>
>> function foo(a::A)
>> for i in 1:length(a.sa)
>> a.sa[i] += 1
>> end
>> end
>>
>> There are a few ways I can break this. 
>>
>> 1. Calling @show on a deepcopy breaks when implemented like this.
>> instance = A(5)
>> @show instance
>> dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
>> being called
>> print(dc.sa) # Works fine
>> @show dc.sa # Works fine
>> @show dc # Seg fault if show is implemented
>>
>> Very strangely, a switch to "println" instead of "print" in the show 
>> function and everything works fine.
>>
>> 2. Even more worrysome, accessing the array in a function (foo):
>> instance = A(5)
>> dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
>> being called
>> foo(instance) # Works fine on the original
>> for i in 1:length(dc.sa)
>> dc.sa[i] += 1
>> end
>> dc.sa # The above worked, and everything prints out fine
>> foo(dc) #Seg fault
>>
>> It turns out, the deepcopy is creating the SharedArray as an Array:
>>
>> julia> instance = A(5)
>> [0.0,0.0,0.0,0.0,0.0]
>>
>> julia> dc = deepcopy(instance);
>>
>> julia> typeof(instance)
>> A
>>
>> julia> typeof(instance.sa)
>> SharedArray{Float64,1}
>>
>> julia> typeof(dc)
>> A
>>
>> julia> typeof(dc.sa)
>> Array{Float64,1}
>>
>> Probably related? Is this a bug or am I doing something silly? Is anyone 
>> aware of a defect open for this or something similar, or should I open a 
>> new one?
>>
>> Cheers,
>> Dan
>>
>

[julia-users] Re: Different result in pmap vs remotecall

[lmagarshack]

That's why I 
push!(thresholds, typemax(eltype(thresholds)))
I was hoping to keep myself from this kind of bounds error...

And that still doesn't explain why it would work in remotecall/normal 
calls, but not pmap, unless I'm missing something.

[julia-users] Re: Different result in pmap vs remotecall

[Daniel Arndt]

You're creating a collection from 0 to 25.

Then you're calling searchsortedfirst on it. Reading the docs explains what 
will happen:

  searchsortedfirst(a, x, [by=,] [lt=,] [rev=false])


  Returns the index of the first value in a greater than or equal to x, 
according to the specified order. Returns length(a)+1 if x is greater than 
all values in a.

Well, there is nothing greater than or equal to 26 in there, so it gives 
you length + 1. Then, you're dereferencing that index which gives you a 
bounds error.

On Monday, 21 December 2015 13:56:48 UTC-4, lmaga...@soasta.com wrote:
>
> First of all thanks for the answer.
>
> And it did work when not adding any workers, so I guess I should updated 
> my code example.
>
> I had called 
> addprocs(1)
>
> before the whole thing.
>
> So I guess now I'm more confused, my *pmap* call works when *nprocs() == 
> 1* but not after adding workers...
> And it's also surprising that I only get that *BoundsError* when my array 
> contains something *>= 26 *(which means *findAndReturn2 *should 
> return the last value of my SortedSet)*.*
>
> eg:
> pmap(findAndReturn2, [1,1,1]) #returns [1,1,1]
> pmap(findAndReturn2, [1,1,26]) #returns [1,1,RemoteException]
>
> Any idea what I could be doing wrong?
>
> Cheers,
> Louis
>

[julia-users] Re: Segmentation fault after deep-copying SharedArray

[Daniel Arndt]

The type of the array being returned seems to be determined by the 
"similar" function:

julia> sa = SharedArray(Float64, 5, init = S -> S[Base.localindexes(S)] = 0)
5-element SharedArray{Float64,1}:
 0.0
 0.0
 0.0
 0.0
 0.0


julia> typeof(similar(sa))
Array{Float64,1}

Perhaps that is the underlying issue.

On Monday, 21 December 2015 14:00:46 UTC-4, Daniel Arndt wrote:
>
> I've been trying to isolate a defect I have in my code for the past couple 
> days, and it seems like SharedArrays have troubles with deepcopying.
>
> Looking at the source, this should be supported (Julia 0.4.2):
>
> https://github.com/JuliaLang/julia/blob/v0.4.2/base/sharedarray.jl#L201
>
> I've created a small stripped down version of the problem I am 
> encountering.
>
> type A
> sa::SharedArray{Float64,1}
> end
>
>
> function A(n::Int)
> sa = SharedArray(Float64, n, init = S -> S[Base.localindexes(S)] = 0)
> return A(sa)
> end
>
> function Base.show(io::IO, a::A)
> print(io,a.sa)
> end
>
> function foo(a::A)
> for i in 1:length(a.sa)
> a.sa[i] += 1
> end
> end
>
> There are a few ways I can break this. 
>
> 1. Calling @show on a deepcopy breaks when implemented like this.
> instance = A(5)
> @show instance
> dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
> being called
> print(dc.sa) # Works fine
> @show dc.sa # Works fine
> @show dc # Seg fault if show is implemented
>
> Very strangely, a switch to "println" instead of "print" in the show 
> function and everything works fine.
>
> 2. Even more worrysome, accessing the array in a function (foo):
> instance = A(5)
> dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
> being called
> foo(instance) # Works fine on the original
> for i in 1:length(dc.sa)
> dc.sa[i] += 1
> end
> dc.sa # The above worked, and everything prints out fine
> foo(dc) #Seg fault
>
> It turns out, the deepcopy is creating the SharedArray as an Array:
>
> julia> instance = A(5)
> [0.0,0.0,0.0,0.0,0.0]
>
> julia> dc = deepcopy(instance);
>
> julia> typeof(instance)
> A
>
> julia> typeof(instance.sa)
> SharedArray{Float64,1}
>
> julia> typeof(dc)
> A
>
> julia> typeof(dc.sa)
> Array{Float64,1}
>
> Probably related? Is this a bug or am I doing something silly? Is anyone 
> aware of a defect open for this or something similar, or should I open a 
> new one?
>
> Cheers,
> Dan
>

[julia-users] Segmentation fault after deep-copying SharedArray

[Daniel Arndt]

I've been trying to isolate a defect I have in my code for the past couple 
days, and it seems like SharedArrays have troubles with deepcopying.

Looking at the source, this should be supported (Julia 0.4.2):

https://github.com/JuliaLang/julia/blob/v0.4.2/base/sharedarray.jl#L201

I've created a small stripped down version of the problem I am encountering.

type A
sa::SharedArray{Float64,1}
end


function A(n::Int)
sa = SharedArray(Float64, n, init = S -> S[Base.localindexes(S)] = 0)
return A(sa)
end

function Base.show(io::IO, a::A)
print(io,a.sa)
end

function foo(a::A)
for i in 1:length(a.sa)
a.sa[i] += 1
end
end

There are a few ways I can break this. 

1. Calling @show on a deepcopy breaks when implemented like this.
instance = A(5)
@show instance
dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
being called
print(dc.sa) # Works fine
@show dc.sa # Works fine
@show dc # Seg fault if show is implemented

Very strangely, a switch to "println" instead of "print" in the show 
function and everything works fine.

2. Even more worrysome, accessing the array in a function (foo):
instance = A(5)
dc = deepcopy(instance); # Semi-colon necessary on REPL to prevent show 
being called
foo(instance) # Works fine on the original
for i in 1:length(dc.sa)
dc.sa[i] += 1
end
dc.sa # The above worked, and everything prints out fine
foo(dc) #Seg fault

It turns out, the deepcopy is creating the SharedArray as an Array:

julia> instance = A(5)
[0.0,0.0,0.0,0.0,0.0]

julia> dc = deepcopy(instance);

julia> typeof(instance)
A

julia> typeof(instance.sa)
SharedArray{Float64,1}

julia> typeof(dc)
A

julia> typeof(dc.sa)
Array{Float64,1}

Probably related? Is this a bug or am I doing something silly? Is anyone 
aware of a defect open for this or something similar, or should I open a 
new one?

Cheers,
Dan

[julia-users] Re: Array view without a single element

[Tero Frondelius]

Hi
Can you use a macro to produce your twenty nested loops? 

[julia-users] Re: Different result in pmap vs remotecall

[lmagarshack]

First of all thanks for the answer.

And it did work when not adding any workers, so I guess I should updated my 
code example.

I had called 
addprocs(1)

before the whole thing.

So I guess now I'm more confused, my *pmap* call works when *nprocs() == 1* but 
not after adding workers...
And it's also surprising that I only get that *BoundsError* when my array 
contains something *>= 26 *(which means *findAndReturn2 *should return the 
last value of my SortedSet)*.*

eg:
pmap(findAndReturn2, [1,1,1]) #returns [1,1,1]
pmap(findAndReturn2, [1,1,26]) #returns [1,1,RemoteException]

Any idea what I could be doing wrong?

Cheers,
Louis

Re: [julia-users] Re: Array view without a single element

[Erik Schnetter]

Tomas

I don't think that what you envision can be implemented efficiently. After
a few steps, there will be an essentially arbitrary mapping between the
current and the original indices. Each time an index is removed, you'll
have to copy O(N) indices, or O(N) array elements.

However, there may be a way out. Does your algorithm come with a pivot
search? Does it already have a re-mapping of indices? If so, you would need
to change the index mapping at every step such that the to-be-removed index
is the last index. Removing the last element from an array is efficient.
This is for example how LAPACK implements dgesv to solve a linear system.

-erik



On Mon, Dec 21, 2015 at 12:19 PM, Matt Bauman  wrote:

> Oh man, fun puzzle.  Even without the allocations, SubArrays that depend
> upon an indexing vector (that is, not computable range) are really quite
> slow since there's a lot of indirection at every element access.
>
> You can solve the all-but-one problem with a custom view type fairly
> easily:
>
> immutable SkipIndexVector{T,A} <: AbstractVector{T}
> data::A
> skip::Int
> end
> SkipIndexVector{A<:AbstractArray}(data::A, skip::Int) = SkipIndexVector{
> eltype(A),A}(data, skip)
> Base.size(S::SkipIndexVector) = (length(S.data)-1,)
> Base.linearindexing{T<:SkipIndexVector}(::Type{T}) = Base.LinearFast()
> Base.getindex(S::SkipIndexVector, i::Int) = S.data[i+(i>=S.skip)]
>
> julia> SkipIndexVector(1:10, 4)
> 9-element SkipIndexVector{Int64,UnitRange{Int64}}:
>   1
>   2
>   3
>   5
>   6
>   7
>   8
>   9
>  10
>
> But if you're recursively constructing views of views, that will quickly
> hit the nested type parameter limit and fall back to dynamic dispatch.
> What if you just make a copy of the array and `pop!` the elements in your
> algorithm?
>
> On Monday, December 21, 2015 at 11:55:45 AM UTC-5, Tomas Lycken wrote:
>>
>> Is there a way to construct a view into an array that skips a single
>> element?
>>
>> For example, to view v except the element at index p, I can do sub(v,
>> [1:p-1;p+1:length(A)]). However, allocating the array with indices is a
>> bottleneck in my code. Is there a way or trick to get rid of it?
>>
>> I’m using this in a recursive function, so there will be subsequent views
>> of views of views, etcetera, and I can’t easily refactor it to be a
>> (nested) loop instead (because I don’t want to hand-code 20 nested loops…)
>>
>> // T
>> ​
>>
>


-- 
Erik Schnetter 
http://www.perimeterinstitute.ca/personal/eschnetter/

[julia-users] Re: Survey: what documentation platforms do you use? Are you happy?

[cormullion]

In the future, perhaps all Julia documentation, both for Base and for 
registered packages, will be in a central location, indexed and 
cross-linked (like Matlab, Mathematica, SciPy, only even better...) Perhaps 
written in a single documentation system written in Julia. I'm looking 
forward to helping out.

[julia-users] Re: Array view without a single element

[Matt Bauman]

Oh man, fun puzzle.  Even without the allocations, SubArrays that depend 
upon an indexing vector (that is, not computable range) are really quite 
slow since there's a lot of indirection at every element access.

You can solve the all-but-one problem with a custom view type fairly easily:

immutable SkipIndexVector{T,A} <: AbstractVector{T}
data::A
skip::Int
end
SkipIndexVector{A<:AbstractArray}(data::A, skip::Int) = SkipIndexVector{
eltype(A),A}(data, skip)
Base.size(S::SkipIndexVector) = (length(S.data)-1,)
Base.linearindexing{T<:SkipIndexVector}(::Type{T}) = Base.LinearFast()
Base.getindex(S::SkipIndexVector, i::Int) = S.data[i+(i>=S.skip)]

julia> SkipIndexVector(1:10, 4)
9-element SkipIndexVector{Int64,UnitRange{Int64}}:
  1
  2
  3
  5
  6
  7
  8
  9
 10

But if you're recursively constructing views of views, that will quickly 
hit the nested type parameter limit and fall back to dynamic dispatch. 
 What if you just make a copy of the array and `pop!` the elements in your 
algorithm?

On Monday, December 21, 2015 at 11:55:45 AM UTC-5, Tomas Lycken wrote:
>
> Is there a way to construct a view into an array that skips a single 
> element?
>
> For example, to view v except the element at index p, I can do sub(v, 
> [1:p-1;p+1:length(A)]). However, allocating the array with indices is a 
> bottleneck in my code. Is there a way or trick to get rid of it?
>
> I’m using this in a recursive function, so there will be subsequent views 
> of views of views, etcetera, and I can’t easily refactor it to be a 
> (nested) loop instead (because I don’t want to hand-code 20 nested loops…)
>
> // T
> ​
>

[julia-users] Array view without a single element

[Tomas Lycken]

Is there a way to construct a view into an array that skips a single 
element?

For example, to view v except the element at index p, I can do sub(v, 
[1:p-1;p+1:length(A)]). However, allocating the array with indices is a 
bottleneck in my code. Is there a way or trick to get rid of it?

I’m using this in a recursive function, so there will be subsequent views 
of views of views, etcetera, and I can’t easily refactor it to be a 
(nested) loop instead (because I don’t want to hand-code 20 nested loops…)

// T
​

[julia-users] Re: Different result in pmap vs remotecall

[Daniel Arndt]

pmap will keep process one as your "delegator", and run the function on a 
worker.

You'll recieve the same result in either of these cases:

1. Don't add any workers, pmap will use process 1. Returns 
9223372036854775807

2. Run the remotecall on something other than process one:
ref = remotecall(findAndReturn2, 2, 26)
Returns:
LoadError: On worker 2:
BoundsError
 in findAndReturn2 at In[1]:8
 in anonymous at multi.jl:904
 in run_work_thunk at multi.jl:645
 in run_work_thunk at multi.jl:654
 in anonymous at task.jl:58
while loading In[1], in expression starting on line 13


 in remotecall_fetch at multi.jl:731
 in call_on_owner at multi.jl:777
 in fetch at multi.jl:795

Cheers,
Dan

On Saturday, 19 December 2015 19:37:15 UTC-4, lmaga...@soasta.com wrote:
>
> Hi all,
>
> I'm using Julia 0.4.2 and I was playing around with pmap and the 
> DataStructures package when I found something weird: the same function call 
> when done through remotecall or pmap gives different results.
>
> Here's a code example:
>
>
> @everywhere using DataStructures
>
>
>
>
>
>
>  
>
> @everywhere global const thresholds = collect(0:25)
>
>  
>
> push!(thresholds, typemax(eltype(thresholds))) #last element is 
> 9223372036854775807
>
>  
>
> @everywhere global const sortedThresholds = SortedSet(thresholds)
>
>
>
>
>  
>
> global const warmup = round(Int64, rand(100) .* 26)
>
>
>
>
>  
>
> @everywhere function findAndReturn2(x::Int64)
>
> return deref((sortedThresholds, searchsortedfirst(sortedThresholds, x
> )))
>
> end
>
>
>
>
> ref = remotecall(findAndReturn2, 1, 26)
>
> fetch(ref) #returns 9223372036854775807
>
>
>
>
> pmap(findAndReturn2, [26]) #returns 
> [RemoteException(2,CapturedException(BoundsError(#undef,#undef),Any[(:findAndReturn2,:none,2,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol(""),-1,1),(:run_work_thunk,symbol("multi.jl"),645,symbol(""),-1,1),(:anonymous,symbol("multi.jl"),907,symbol("task.jl"),63,1)]))]
>
>
>
> I would expect both calls to return the same thing (albeit wrapped in an 
> Array in the case of pmap and not in the case of remotecall)
>
>
> Cheers,
>
> Louis
>
>

[julia-users] Re: Survey: what documentation platforms do you use? Are you happy?

[Ken B]

I like to use Markdown with latex for maths. 

For a project earlier this year I've used a markdown extension to host it 
on readthedocs with mathjax support.
It was a bit tricky to set up and doesn't feel so robust, but it works. 
Here's what I 
did: https://github.com/mkdocs/mkdocs/issues/253#issuecomment-97424632

I've seen some Julia projects that seem to automatically convert docstrings 
to documentation but I prefer to use the docstrings only for REPL and write 
out the documentation in markdown.

On Monday, 21 December 2015 06:52:59 UTC+1, Jeffrey Sarnoff wrote:
>
> Whatever you decide, pandoc  should help you automate 
> translation, and it is strong enough to lean on. I have not used a markup 
> system with readable raw files and good math support and just enough 
> flexibility.
>
>
> On Sunday, December 20, 2015 at 1:43:35 PM UTC-5, Tomas Lycken wrote:
>>
>> Over at Interpolations.jl, we've started thinking about restructuring our 
>> documentation a little. We basically have (or will have) three levels of 
>> documentation: usage docs, specifying how to use the library; math docs, 
>> fleshing out the mathematical background to the algorithms and assumptions 
>> we've used; and devdocs, which describe implementation details that might 
>> not be obvious just from reading the code (for one thing, the library is 
>> quite heavy on metaprogramming).
>>
>> The usage/api docs are slowly being migrated to docstrings, with the hope 
>> that the users will actually be able to find it :)
>>
>> I'm looking for input on what tools, formats, platforms etc have worked 
>> well for the other two types of documentation in other projects, and in 
>> what form you, as users or contributors to the library, would prefer 
>> consuming that documentation.
>>
>> A stong requirement is that it is editable, and preferrably also quite 
>> readable, as plaintext. For the math docs, it's also important (obviously) 
>> that the support for rendering equations, matrices etc is good.
>>
>> Have you used any tools or platforms previously that solve these problems 
>> well? Do you have recommendations about which ones to avoid?
>>
>> All suggestions are welcome!
>>
>> // T
>>
>