[julia-users] Re: Extending functions in Base (or another module)

[Pablo Zubieta]

You might also want to express your perspective on why a function such as 
invoke is needed here https://github.com/JuliaLang/julia/pull/13123.

On Tuesday, June 28, 2016 at 5:26:16 PM UTC+2, Bill Hart wrote:
>
> We have hit an issue that we can't seem to find a workaround for. Our only 
> working workaround is no longer supported by Julia 0.5.
>
> The issue
> 
>
> We implement determinant for various kinds of matrix types in Nemo and 
> Hecke (two computer algebra/number theory packages). To do this, we extend 
> Base.det in Nemo.
>
> Hecke depends on Nemo and tries to extend Nemo.det (or Base.det).
>
> Hecke wants to define det for one of its own types, let's call it 
> SpecialMat. Now Hecke's SpecialMat belongs to Nemo's abstract type MatElem, 
> i.e. SpecialMat <: MatElem.
>
> Nothing unusual so far.
>
> The problem is: Hecke would like to call the det function provided by Nemo 
> in case the algorithm they provide is going to be slower (a decision that 
> is made at runtime).
>
> What we try
> =
>
> So what we naturally try in Hecke is something like the following 
> (obviously this code doesn't actually work):
>
> module Hecke
>
> using Nemo
>
> type SpecialMat <: MatElem  ## Nemo has a MatElem type class and a 
> "generic" det algorithm for any type that belongs to it
># some data
> end
>
> function det(m::SpecialMat)
>
> # do different things depending on properties of the matrix m
>
>if check_some_properties_of_a_matrix(m)
># implementation of special determinant algorithm that only works 
> for Hecke SpecialMat's
>else
>Nemo.det(m) # fallback to the Nemo implementation of det (which is 
> extensive)
>end
> end
>
> export det
>
> end # module
>
> Here are some potential solutions we tried which didn't work:
>
> 1) Only import the functions from Nemo that Hecke doesn't need to 
> overload, i.e. don't import det from Nemo (or Base)
>
> This causes Julia to tell the user that det could refer to Base.det or 
> Hecke.det, even though Base.det doesn't provide a function for the 
> specified type. We certainly can't expect the user to have to look up all 
> the documentation for Base, Nemo and Hecke every time they call a function 
> so they know how to qualify it. So this isn't a workable solution, 
> obviously. It's also far too verbose.
>
> 2) Try to qualify the function name with the name of the module, i.e. call 
> Nemo.det in the Hecke definition of the function, as above. 
>
>This doesn't work, since it is Nemo.det that currently has being 
> overloaded. So the Hecke det function just segfaults when called.
>
> 3) Look up the method table for the function we require and call the 
> specific method. 
>
>This works in Julia 0.4, but the ability to call methods has been 
> removed in 0.5.
>
> This is an exceedingly frustrating problem. In fact it also occurs within 
> Nemo itself, since every time we want to implement a specialised version of 
> a generic function for a specific type, and have it fall back to the 
> generic version in certain cases determined at runtime, we can't do it, 
> without first renaming the generic implementation to something else with a 
> different name.
>
> This sort of thing is making it very difficult to build large systems. 
> It's not fair to the developers of Hecke to ask them to duplicate all the 
> code in Nemo just so they can make this work, or alternatively force Nemo 
> to define every function twice so that there is a callable version with a 
> different name.
>
> Does anyone know a workaround (any hack that works will do) for this 
> issue, that works in Julia 0.4 and Julia 0.5?
>
> And is there a plan to fix this sort of issue in Julia in the future? The 
> module system currently makes it quite hard to work with multiple modules. 
> We are often encouraged to split our large systems into smaller 
> modules/packages to get around certain issues, and then when we do that, 
> the module system actually gets in the way.
>
> Bill.
>

Re: [julia-users] Re: Extending functions in Base (or another module)

[Mauro]

There is some discussion on invoke on github:
https://github.com/JuliaLang/julia/pull/13123
maybe you want to weight in.

On Tue, 2016-06-28 at 19:19, 'Bill Hart' via julia-users 
 wrote:
> You are a life saver. This is *precisely* what we need. Thank you for
> solving a very difficult problem for us. We were really pulling our hair
> out after searching for a solution.
>
> julia> module Nemo
>   import Base: det
>   abstract MatElem
>   function det(a::MatElem)
>  return 1
>   end
>   type nemomat <: MatElem
>   end
>   export nemomat
>   export det
>end
> Nemo
>
> julia> module Hecke
>   using Nemo
>   import Nemo.det
>   type SpecialMat <: Nemo.MatElem
>  data::Int
>   end
>   function det(a::SpecialMat)
>  if a.data == 4
> return 3
>  else
> return invoke(det, (Nemo.MatElem,), a)
>  end
>   end
>   export SpecialMat
>   export det
>end
> Hecke
>
> julia> using Hecke
>
> julia> s = SpecialMat(3)
> Hecke.SpecialMat(3)
>
> julia> t = SpecialMat(4)
> Hecke.SpecialMat(4)
>
> julia> det(s)
> 1
>
> julia> det(t)
> 3
>
> Let's help search engines with this, since we were unable to find anything,
> and it is such an important issue:
>
> trouble extending a Base function in Julia
> How do I call a specific version of a function in Julia for specific types
> How do I call a more general version of a function in Julia
> How do I call a less specific version of a function in Julia
> How do I call a specific method in Julia
> method to apply a function in Julia
> invoking a given version of a function in Julia
>
> To the Julia devs: please, please don't remove this functionality!
>
> Bill.
>
> On Tuesday, 28 June 2016 18:40:27 UTC+2, Rafael Fourquet wrote:
>>
>> I'm far from expert on those questions, but would the "invoke" function
>> work?
>> I think it's considered to be a tool of the last resort, but seems to
>> be the situation you are in!
>>
>> invoke(f, (types...), args...)
>>
>>   Invoke a method for the given generic function matching the
>> specified types (as a tuple), on the specified arguments. The
>> arguments must be compatible with
>>   the specified types. This allows invoking a method other than the
>> most specific matching method, which is useful when the behavior of a
>> more general
>>   definition is explicitly needed (often as part of the implementation
>> of a more specific method of the same function).
>>

Re: [julia-users] Re: Extending functions in Base (or another module)

['Bill Hart' via julia-users]

You are a life saver. This is *precisely* what we need. Thank you for 
solving a very difficult problem for us. We were really pulling our hair 
out after searching for a solution.

julia> module Nemo
  import Base: det
  abstract MatElem
  function det(a::MatElem)
 return 1
  end
  type nemomat <: MatElem
  end
  export nemomat
  export det
   end
Nemo

julia> module Hecke
  using Nemo
  import Nemo.det
  type SpecialMat <: Nemo.MatElem
 data::Int
  end
  function det(a::SpecialMat)
 if a.data == 4
return 3
 else
return invoke(det, (Nemo.MatElem,), a)
 end
  end
  export SpecialMat
  export det
   end
Hecke

julia> using Hecke

julia> s = SpecialMat(3)
Hecke.SpecialMat(3)

julia> t = SpecialMat(4)
Hecke.SpecialMat(4)

julia> det(s)
1

julia> det(t)
3

Let's help search engines with this, since we were unable to find anything, 
and it is such an important issue:

trouble extending a Base function in Julia
How do I call a specific version of a function in Julia for specific types
How do I call a more general version of a function in Julia
How do I call a less specific version of a function in Julia
How do I call a specific method in Julia
method to apply a function in Julia
invoking a given version of a function in Julia

To the Julia devs: please, please don't remove this functionality!

Bill.

On Tuesday, 28 June 2016 18:40:27 UTC+2, Rafael Fourquet wrote:
>
> I'm far from expert on those questions, but would the "invoke" function 
> work? 
> I think it's considered to be a tool of the last resort, but seems to 
> be the situation you are in! 
>
> invoke(f, (types...), args...) 
>
>   Invoke a method for the given generic function matching the 
> specified types (as a tuple), on the specified arguments. The 
> arguments must be compatible with 
>   the specified types. This allows invoking a method other than the 
> most specific matching method, which is useful when the behavior of a 
> more general 
>   definition is explicitly needed (often as part of the implementation 
> of a more specific method of the same function). 
>

[julia-users] Re: Extending functions in Base (or another module)

['Bill Hart' via julia-users]

Another possibility: introduce a special "extends" syntax to Julia, e.g.

module Hecke

type SpecialMat <: Nemo.MatElem
end

function det(a::SpecialMat)
   if blah
   # do whatever
   else
  Nemo.det
   end
end

extends det Nemo.det # tells Julia to somehow treat the det implementation 
above specially so it can call the Nemo version of the function without 
calling itself.

end # module

The behaviour would be: if extends is missing, it gives an ambiguity 
warning about Nemo.det vs Hecke.det if someone calls det after including 
Hecke and Nemo.

Bill.

[julia-users] Re: Extending functions in Base (or another module)

['Bill Hart' via julia-users]

By the way, here are some potential things that could solve this for us:

1) Allow individual methods to be callable, as in Julia 0.4 (so long as we 
can look the methods up at "compile time" in a module and assign them to a 
const in the module so that they can be called). As far as we can see, this 
has been removed in Julia 0.5.

2) Have a syntax for calling a method with a given signature, e.g. 
apply(det, (MatElem,), m) or something like that. Something like this used 
to exist in Julia but has been removed.

3) Have Julia not print an ambiguity warning about 
Base.det/Nemo.det/Hecke.det when calling det on a matrix where there is a 
specific implementation for that type in Hecke, but only something more 
general in the other modules. Currently Julia complains that you might mean 
Base.det or Hecke.det even though the former does not implement a function 
for matrices of type SpecialMat. But why? What's ambiguous here?

4) Remove the requirement for a module to extend the definition from Base 
or whatever other packages it uses, if it is only defining the function for 
one of its own types and not a type that is present in Base or that other 
module.

5) Introduce a way of calling an implementation provided by a specific 
module, e.g. specifying Nemo.det would only call the version actually 
defined in Nemo. This might feel like the current behaviour, but it doesn't 
actually work, since it is Base.det or Nemo.det that Hecke needs to be 
overloaded in the first place. So calling Nemo.det from within Hecke.det 
just causes it to call Hecke.det for that specific type, causing a stack 
overflow. 

6) Allow us to define function Hecke.det (and still export it as det) in 
Hecke, and call Nemo.det from within that implementation. Again, this might 
feel like the current behaviour, but it doesn't work because it is Nemo.det 
or Base.det that needs to be defined in Hecke, currently.

Bill.