Hi, In my view there are two related but separate issues: (1) What should we do for 3.1 release, and (2) How do we want to design the optimization package for the future, such that it is easily extended to linear and non-linear constraints, sparse matrix operations, etc.
In regards to (1), I feel that designing a proper API for the future is a very important problem that would take a while, to not only implement and test, but to also come to an agreement. This future API should not be limited by the backwards capability with 3.0, but should instead be careful designed from the ground up. That is why I propose that we do not let these issues set back the much needed release of 3.1. For example in 3.0 CMAES bound constraints are actually broken. This should be patched ASAP. There are new features. While my combined function issue is important, it is a new feature, and by no means critical. In my view it is better to leave the Function classes as they are (or at least similar to what they are), so the user is comfortable with the interface. In regards to (2), Here are some things that we need to think about 1) How to properly design the function interfaces for the optimization packages, such that they address my combined value and gradient computation, but also have proper OO structure such that a function that is used in a "value+gradient" optimization can also be used for a no-gradient direct optimization? What wrapper classes should we provide for the user? Should this functions be also comparable with analysis package (or some other package), or would optimization performance suffer? How should these values be stored so they can be efficiently used by the optimizer without significant amount of memory copies. 2) Changing the return class for "optimization" function. Like I mentioned before, I don't think the optimization class should keep state on the previous optimization. All RMS, chi^2 values are part of the optimization result and not part of the optimizer. An example of such use is where a user computed values from different initial points and then compares results. 3) It might *not* be a good idea to have constraints be given as part of options into an "optimization" function. First, different implementations have different limitations, so one method might take non-linear constraints, another doesn't. This should be explicitly visible through the constructors of the class. Second, linear constraints,and probably other constrains require pre-processing before use. For example, if you have linear equality constraints in an optimization, you need to extract the null space using QR or SVD. If you have a lot of constraints this process is expensive, and should be amortized for calls with same constrains but different initial points. There are probably other examples of this. We should think and discuss this, before jumping to a design. It is very unfortunate that there is a lack of man power to improve the algorithms. I am very sad to say, but I have barely time to even reply to these emails. I look forward to hearing what other people think, Konstantin On Dec 3, 2012, at 8:16 AM, Gilles Sadowski <gil...@harfang.homelinux.org> wrote: > On Sat, Dec 01, 2012 at 04:58:30PM -0500, Konstantin Berlin wrote: >>> >>> I would propose to simply revert my changes on the optimization package >>> and prepare for a reorganization for 4.0. I understand I focused only on >>> the type of problems Gilles and myself routinely use, i .e. small size >>> problems >>> where the cost of the evaluation is several orders of magnitude larger than >>> the >>> data copying. I forgot the dual case with very large data sets. I >>> apologize for that. >>> >>> When 3.1 will be out, we will have to solve this so both cases are handled >>> efficiently, >>> and this would probably be implemented in 4.0. >>> >>> Does this seems reasonable? >>> >>> Best regards >>> Luc >>> >> >> Don't blame yourself, its hard to think of all possible uses. > > Indeed. And it is also hard to get a design right without being guided by > actual use-cases. > Thus... [see below] > >> I think reverting back to original interface is the best choice for 3.1. > > I don't understand (cf. my other post); in the "orignal" interface (i.e. > "DifferentiableMultivariateVectorFunction"), you have exactly the same > problem as with what Luc proposed earlier: 2 separate functions, one for > the value and one for the Jacobian. > Please indicate whether my proposal is better from this point-of-view: One > function call will return an array of "value and gradient" objects, and > references to those can be accessed directly inside the optimizer's code > to get the Jacobian data, without copying. > >> In the future it is worth finding and adding "standard" industry benchmarks >> to the tests. > > ... this issue came up several times, but was not able to gather the > necessary "human-power": see > https://issues.apache.org/jira/browse/MATH-678 > and > https://issues.apache.org/jira/browse/MATH-763 > >> >> Speaking of optimizers... Would it not be better to move all the getRMS, >> getChiSquared into >> the return class, along with all other information, like how many iterations >> it took, function >> evaluations, etc? That is how I find it more natural to use, so I end up >> creating a wrapper. >> These properties are not properties of the optimizer but are properties of >> the specific >> optimization evaluation of the function with the specific initial value. > > As always, patches are welcome that practically demonstrate the improvement. > > > Regards, > Gilles > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
