[julia-users] Re: When does colon indexing get evaluated / converted?

['Greg Plowman' via julia-users]

Matt,
Thankyou so much for your great reply.
And yes it does make sense now after such a helpful and well-targeted 
explanation.
You explained "colon-lowering" in a "explanation-lowered" way that a 
non-expert can understand.
This is refreshingly thoughtful and helpful.
Thanks again.
-- Greg

 
On Tuesday, September 22, 2015 at 11:09:41 PM UTC+10, Matt Bauman wrote:

> A colon by itself is simply a synonym for `Colon()`, both on 0.3 and 0.4:
>  
> julia> :
> Colon()
>  
> You can use colons in normal function calls and dispatch in both 0.3 and 
> 0.4:
>  
> julia> f(x::Colon) = x
>f(:)
> Colon()
>  
> That's what's happening with that sub definition.  `sub` acts like 
> indexing, but it's just a normal function call.  And so the Colon() passes 
> through unchanged in both 0.3 and 0.4.
>  
> Indexing expressions are different.  There are some special lowering steps 
> to handle colon (just in 0.3) and end (in both 0.3 and 0.4).  "Lowering" is 
> an intermediate step between parsing and compilation, and you can see what 
> happens to an expression during lowering by calling `expand`.  The expanded 
> expressions have some funny things in them (`top(endof)(A)` is a special 
> internal pseudo-syntax that is kinda-sorta like saying `Base.endof(A)`), 
> but you can see how the `end` keyword lowers to getindex calls in both 0.3 
> and 0.4:
>  
> julia> expand(:(A[end]))
> :(getindex(A,(top(endof))(A)))
>  
> julia> expand(:(A[1, end]))
> :(getindex(A,1,(top(trailingsize))(A,2)))
>  
> julia> expand(:(A[1, end, 3]))
> :(getindex(A,1,(top(size))(A,2),3))
>  
> So, now to the difference between 0.3 and 0.4: the special lowering step 
> for colons in indexing expressions was removed.
>  
> # Julia 0.3:
> julia> expand(:(A[:]))
> :(getindex(A,colon(1,(top(endof))(A
>  
> # Julia 0.4:
> julia> expand(:(A[:]))
> :(getindex(A,:))
>  
> You can see that in 0.3, it effectively expanded to `1:end`.  But 0.4 just 
> left the `:` symbol as it was, and it just gets evaluated normally as an 
> argument to getindex.  It's no longer special. So in 0.4, you can 
> specialize dispatch on Colon indices.
>  
> Make sense?
>  
> On Tuesday, September 22, 2015 at 1:07:01 AM UTC-4, Greg Plowman wrote:
>>
>> Hi All,
>> Thanks all for your replies.
>>  
>> OK I can see this will be much easier in v0.4.
>> I will revisit when v0.4 released.
>>  
>> I'm still curious about colon and end
>>
>>> Colon lowering changed in 0.4, 
>>
>>  
>> Matt, could you expand on this? How/when is this done in v0.3 vs v0.4?
>>  
>> Does this mean v0.3 code attempting to dispatch on Colon type cannot 
>> work? (for example the code from subarray.jl quoted below) 
>>  
>> sub(A::AbstractArray, I::Union(RangeIndex, Colon)...) = sub(A, ntuple(
>> length(I), i-> isa(I[i], Colon) ? (1:size(A,i)) : I[i])...)
>>  
>> I noticed that  OffsetArrays (https://github.com/alsam/OffsetArrays.jl 
>> 
>>  
>> jA 
>> )
>>  
>> defines const (..) = Colon(), presumably to use .. in place of :
>>  
>> Will it work in v0.4?
>> How and when is end converted?
>>  
>> Thanks again,
>> Greg
>>  
>>  
>>  
>>
>

[julia-users] Re: When does colon indexing get evaluated / converted?

[Matt Bauman]

A colon by itself is simply a synonym for `Colon()`, both on 0.3 and 0.4:

julia> :
Colon()

You can use colons in normal function calls and dispatch in both 0.3 and 
0.4:

julia> f(x::Colon) = x
   f(:)
Colon()

That's what's happening with that sub definition.  `sub` acts like 
indexing, but it's just a normal function call.  And so the Colon() passes 
through unchanged in both 0.3 and 0.4.

Indexing expressions are different.  There are some special lowering steps 
to handle colon (just in 0.3) and end (in both 0.3 and 0.4).  "Lowering" is 
an intermediate step between parsing and compilation, and you can see what 
happens to an expression during lowering by calling `expand`.  The expanded 
expressions have some funny things in them (`top(endof)(A)` is a special 
internal pseudo-syntax that is kinda-sorta like saying `Base.endof(A)`), 
but you can see how the `end` keyword lowers to getindex calls in both 0.3 
and 0.4:

julia> expand(:(A[end]))
:(getindex(A,(top(endof))(A)))

julia> expand(:(A[1, end]))
:(getindex(A,1,(top(trailingsize))(A,2)))

julia> expand(:(A[1, end, 3]))
:(getindex(A,1,(top(size))(A,2),3))

So, now to the difference between 0.3 and 0.4: the special lowering step 
for colons in indexing expressions was removed.

# Julia 0.3:
julia> expand(:(A[:]))
:(getindex(A,colon(1,(top(endof))(A

# Julia 0.4:
julia> expand(:(A[:]))
:(getindex(A,:))

You can see that in 0.3, it effectively expanded to `1:end`.  But 0.4 just 
left the `:` symbol as it was, and it just gets evaluated normally as an 
argument to getindex.  It's no longer special. So in 0.4, you can 
specialize dispatch on Colon indices.

Make sense?

On Tuesday, September 22, 2015 at 1:07:01 AM UTC-4, Greg Plowman wrote:
>
> Hi All,
> Thanks all for your replies.
>
> OK I can see this will be much easier in v0.4.
> I will revisit when v0.4 released.
>
> I'm still curious about colon and end
>
>> Colon lowering changed in 0.4, 
>
>
> Matt, could you expand on this? How/when is this done in v0.3 vs v0.4?
>
> Does this mean v0.3 code attempting to dispatch on Colon type cannot 
> work? (for example the code from subarray.jl quoted below) 
>
> sub(A::AbstractArray, I::Union(RangeIndex, Colon)...) = sub(A, ntuple(
> length(I), i-> isa(I[i], Colon) ? (1:size(A,i)) : I[i])...)
>
> I noticed that  OffsetArrays (https://github.com/alsam/OffsetArrays.jl 
> 
>  
> jA 
> )
>  
> defines const (..) = Colon(), presumably to use .. in place of :
>
> Will it work in v0.4?
> How and when is end converted?
>
> Thanks again,
> Greg
>
>
>
>

[julia-users] Re: When does colon indexing get evaluated / converted?

['Greg Plowman' via julia-users]

Hi All,
Thanks all for your replies.

OK I can see this will be much easier in v0.4.
I will revisit when v0.4 released.

I'm still curious about colon and end

> Colon lowering changed in 0.4, 


Matt, could you expand on this? How/when is this done in v0.3 vs v0.4?

Does this mean v0.3 code attempting to dispatch on Colon type cannot work? 
(for example the code from subarray.jl quoted below) 

sub(A::AbstractArray, I::Union(RangeIndex, Colon)...) = sub(A, ntuple(length
(I), i-> isa(I[i], Colon) ? (1:size(A,i)) : I[i])...)

I noticed that  OffsetArrays (https://github.com/alsam/OffsetArrays.jl 

 
jA 
)
 
defines const (..) = Colon(), presumably to use .. in place of :

Will it work in v0.4?
How and when is end converted?

Thanks again,
Greg

[julia-users] Re: When does colon indexing get evaluated / converted?

[Matt Bauman]

Here's a quick rundown on the status quo:

* Colon lowering changed in 0.4, as did the AbstractArray's interface. 
 It's *much* easier to make your own arrays in 0.4.  I suggest updating to 
the latest RC.

* There is still a very strong assumption in base Julia that the valid 
indices in an AbstractArray `A` for dimension `d` are in `1:size(A, d)`.
* This means that once a base method has ensured that all the indices it 
visits are within `1:size(A,d)`, it might wrap its algorithm with 
`@inbounds`.
* This means that if you break that assumption, using a base function could 
result in the wrong answer and/or memory corruption and/or segfaults.
* It also means that the `end` keyword won't lower correctly for your array

Now, it is possible to break this assumption and still have things work, 
but for right now, it's undocumented magic.  There's several of us very 
interested in doing this, so it'll get better.  Also new in 0.4 is the 
eachindex method, which helps here, too.  The key is to aggressively check 
bounds and explicitly error on any indexing method that you know might be 
ambiguous in its meaning (e.g., does the caller know that your array has 
funny indexing semantics or not?).

Related 
discussion: 
https://groups.google.com/forum/#!searchin/julia-users/offsetarray%7Csort:date/julia-users/fNisYpMdZ6o/tahp5AXaAwAJ

On Sunday, September 20, 2015 at 9:34:24 PM UTC-4, Greg Plowman wrote:
>
> Hi,
>
> I'm trying to define a custom Array type that can be indexed using 
> arbitrary ranges.
>
> e.g. A = MyArray(Int, 3:8) would define a 6-element vector with indexes 
> ranging from 3 to 8, rather than the default 1 to 6.
>
> I've made some progress, but am now stuck on how to handle colon indexing.
>
> A[4:6] works by defining appropriate getindex and setindex!
>
> e.g.  setindex!{T,S<:Real}(A::MyArray{T,1}, value, I::AbstractVector{S}) 
> = ...
>
> but A[:] = 0 seems to get translated to A[1:6] before dispatch on 
> setindex!, so I can't hijack the call.
>
> From subarray.jl, the code below suggests I can specialise on the Colon 
> type, but this doesn't seem to work for me. Colon appears to be converted 
> to UnitRange *before* calling setindex!
>
> sub(A::AbstractArray, I::Union(RangeIndex, Colon)...) = sub(A, ntuple(
> length(I), i-> isa(I[i], Colon) ? (1:size(A,i)) : I[i])...)
>
>
> Is there a way around this?
> Should I be able to specialise on the colon argument?
>
> -- Greg
>

Re: [julia-users] Re: When does colon indexing get evaluated / converted?

[Tim Holy]

In julia 0.5 I plan to add eachindex(A, d) which can return the appropriate 
range for your object along axis d.

--Tim

On Monday, September 21, 2015 06:20:02 AM Matt Bauman wrote:
> Here's a quick rundown on the status quo:
> 
> * Colon lowering changed in 0.4, as did the AbstractArray's interface.
>  It's *much* easier to make your own arrays in 0.4.  I suggest updating to
> the latest RC.
> 
> * There is still a very strong assumption in base Julia that the valid
> indices in an AbstractArray `A` for dimension `d` are in `1:size(A, d)`.
> * This means that once a base method has ensured that all the indices it
> visits are within `1:size(A,d)`, it might wrap its algorithm with
> `@inbounds`.
> * This means that if you break that assumption, using a base function could
> result in the wrong answer and/or memory corruption and/or segfaults.
> * It also means that the `end` keyword won't lower correctly for your array
> 
> Now, it is possible to break this assumption and still have things work,
> but for right now, it's undocumented magic.  There's several of us very
> interested in doing this, so it'll get better.  Also new in 0.4 is the
> eachindex method, which helps here, too.  The key is to aggressively check
> bounds and explicitly error on any indexing method that you know might be
> ambiguous in its meaning (e.g., does the caller know that your array has
> funny indexing semantics or not?).
> 
> Related
> discussion:
> https://groups.google.com/forum/#!searchin/julia-users/offsetarray%7Csort:d
> ate/julia-users/fNisYpMdZ6o/tahp5AXaAwAJ
> On Sunday, September 20, 2015 at 9:34:24 PM UTC-4, Greg Plowman wrote:
> > Hi,
> > 
> > I'm trying to define a custom Array type that can be indexed using
> > arbitrary ranges.
> > 
> > e.g. A = MyArray(Int, 3:8) would define a 6-element vector with indexes
> > ranging from 3 to 8, rather than the default 1 to 6.
> > 
> > I've made some progress, but am now stuck on how to handle colon indexing.
> > 
> > A[4:6] works by defining appropriate getindex and setindex!
> > 
> > e.g.  setindex!{T,S<:Real}(A::MyArray{T,1}, value, I::AbstractVector{S})
> > = ...
> > 
> > but A[:] = 0 seems to get translated to A[1:6] before dispatch on
> > setindex!, so I can't hijack the call.
> > 
> > From subarray.jl, the code below suggests I can specialise on the Colon
> > type, but this doesn't seem to work for me. Colon appears to be converted
> > to UnitRange *before* calling setindex!
> > 
> > sub(A::AbstractArray, I::Union(RangeIndex, Colon)...) = sub(A, ntuple(
> > length(I), i-> isa(I[i], Colon) ? (1:size(A,i)) : I[i])...)
> > 
> > 
> > Is there a way around this?
> > Should I be able to specialise on the colon argument?
> > 
> > -- Greg