MathProgBase was designed most carefully out of these types of systems. All solvers implement the same (well, one or more of a small number) abstract interface. MathProgBase defines that interface in terms of naming and API for consumers and providers, and solver packages define concrete instances of that interface. That way consumers (JuMP, Convex.jl, etc) can use multiple backends via the same interface, without having to depend on all of them.
If you can't fit your code structure to match this abstract interface design, or some form of totally optional dependency injection (like the way JSON.jl can use OrderedDict from DataStructures just by passing in OrderedDict as an optional type argument to some functions), then you need to be very careful how your implementation interacts with precompilation. Requires interacts badly, as do other similar implementations (try-catches like Compose.jl uses, the @extern macro in Extern.jl, the @glue macro seen in several packages, etc). If the list of which conditional dependencies are available at precompile time differs from the list that are available at runtime, then you will either get reduced functionality (if more became available after precompile time) or errors (if some of the conditional dependencies were removed after precompile time). I haven't seen this done yet in a careful way where the list of which conditional dependencies are available is saved properly into the precompile information, then checked at runtime so the module is correctly recompiled when the list of available conditional dependencies changes. Whatever version eventually gets implemented in the base language will have to tie into precompilation in this way. For now, a reliable way of structuring conditional dependencies is splitting your code into separate packages. For example, the author of GLPK.jl did not want to add a dependency (conditional or otherwise) on MathProgBase.jl, so the combination GLPK + MathProgBase code is implemented in GLPKMathProgInterface.jl which depends on both. It's a bit of hassle to split your code up into separate packages, but it works correctly without you having to play any tricks. On Tuesday, July 12, 2016 at 8:40:05 PM UTC-7, Chris Rackauckas wrote: > > Hey, > I wanted to look more into using conditional dependencies, and am > looking for insight from both developers who have done this successfully > and feedback from users on what is more intuitive. > > Just so we're all on the same page, a conditional dependency is a > package which your package does not need in most cases. One big example is > Plots.jl which allows you to use different backends (PyPlot.jl, GR.jl, > etc.) for your plotting. As you can see here > <https://github.com/tbreloff/Plots.jl/blob/master/src/backends/gr.jl> there > is an eval code which will import something like the GR backend during some > initialization step the first time GR is needed. Another way this is used > is somewhere like JuMP + friends. In this ecosystem, where (specifically > for MathProgBase.jl) solvers may be imported by users > <http://mathprogbasejl.readthedocs.io/en/latest/solvers.html#choosing-solvers> > and > then the appropriate functions are passed in. From my understanding, this > means that the definitions of the abstract types must be in a different > package, which the semi-dependencies must import and extend? It seems that > there is another system here > <https://github.com/JuliaOpt/MathProgBase.jl/blob/master/src/defaultsolvers.jl> > where > some things are actually imported to MathProgBase? (I am wondering if > someone who knows this code could explain a little bit what's going on > here). > > In DifferentialEquations.jl, I am using a third method for doing this > via directly looking towards the Main module. An example is in the sde > solvers > <https://github.com/ChrisRackauckas/DifferentialEquations.jl/blob/master/src/sde/sdeSolvers.jl> > where > if a boolean is given and isdefined(Main,:Atom)==true, then Juno/Atom's > progress bar is used by the solver by calling > > (atomLoaded && progressBar) ? Main.Atom.progress(t/T) : nothing #Use > Atom's progressbar if loaded > > every so often in the loop. This method for a conditional import requires, > like in the MathProgBase.jl implementation with passing the solver, that > the user explicitly put "using Atom" at the top of the code for it to work. > Luckily Juno does this automatically so this ends up seemless. > > I am wondering what the pros/cons of these different methods for > implementing conditional dependencies are in order to start "ballooning" my > feature list, while not tagging on a large number of dependencies. There > are two different places where I see new conditional dependencies being > added in my code. One place is with "new solver methods". As you can see > from something like the ODE solvers > <https://github.com/ChrisRackauckas/DifferentialEquations.jl/blob/master/src/ode/odeSolvers.jl>, > > I allow a user to choose a large number of different ways of solving the > method. I plan on, in the very near future, making it conditional -> loop > instead of having the conditional in the loop (that was just because of how > I first wrote it!), and once that's the case, the next goal is to add other > ODE solvers from other people's packages, i.e. make a new part of the > conditional "if alg==:ode23" call ODE.jl's ode23 on the ODE. In this case, > the dependency may be pretty obvious to the user, maybe it's better to > require that the user do "using ODE" in their code and make it explicit? > The Plots.jl method would have it silently import ODE and throw a good > error message if it doesn't exist. What are the pros/cons here? > > However, a very different case for adding new conditional dependencies > is something like seen in the implementation of the trapezoidal method. The > loop for the solver for this is simply: > > elseif alg=="Trapezoid" > uOld = copy(u) > u = vec(u) > nlres = nlsolve((u,resid)->rhs(u,resid,uOld,t,Δt),u) > u = reshape(nlres.zero,sizeu...) > > > i.e. it just solves an appropriate implicit equation using NLsolve.jl. > Because of this (and a few other solvers which use an implicit solver), > DifferentialEquations.jl currently depends on NLsolve, though in many > (maybe most) cases the user won't need it. A simple implementation by using > Main.nlsolve would require the user to have "using NLsolve" at the top of > their code which may scare a lot of people away (thinking there is a bug > instead of a missing package), so I think a Plots.jl implementation of > silently importing the package when needed and throwing an error if it > doesn't exist would be most suitable? > > I am quite interested in what people think about this. I wonder if > anything is really becoming "standard" for implementing these kinds of > conditional dependencies and will follow whatever standard makes sense to > both users and other developers (Stefan mentions this as something that > may be coming later in Julia, so maybe we should starting thinking about > the best way to do this now! <https://www.youtube.com/watch?v=5gXMpbY1kJY>). > [And sorry for the current state of the DifferentialEquations.jl code base, > it was written to work first, and I know it needs some cleaning/re-naming, > and slight re-structuring of the main loop to put the conditionals out...] >
