While I think the general concept is a good one, I feel that any implementation would be somewhat arbitrary.
(Brace yourself for a long winded late-night rant about parameters) For instance, changing one parameter by 10 might be "equivelent" to changing another parameter by 20. I don't even know how you would quantify the "range" except for the cases where you can determine the full range in which the parameter applies (i.e. the range in which a certain parameter can be that will generate at least N trades). Then you have to cut this range up to compare it, somehow, and I think that the ideal distribution would be very nonlinear. Furthermore, you simply cannot judge performance with a number. Surface plots are a must. My main beef with the JBT plots is that you only get to see the "best" and "worst" strategies for a range of 2 variables. It would be infinitely more valuable to look at a plot of X vs Y while Z=30 and Q=40. This, in my opinion, is an absolute must for strategies with more than 2 parameters. Why? Often times you need to weed out parameters that don't do much. Flexibility in your plots gives you the ability to identify such parameters. Armed with some exploration you can significantly shrink your datasets. Which brings me to my last point: you really need to cut down the number of parameters if you have more than 4. I find 4 to be too much. Some might think that this is not possible, some strategies may need these parameters, or that not exploring them could be dangerous. Well, I present you with another theory. EVERY strategy has infinite parameters, but you are fixing all but a few to a constant value. Too many parameters does leads to bad strategies. The main problem is that you start "curve fitting". You can always make a curve fit the data - but if there is no guarantee that the shape will hold up in the future. Keep the strategy basic, based upon a trading theory, and keep the number of parameters low. On Wed, Aug 12, 2009 at 10:25 PM, nonlinear5 <[email protected]>wrote: > > I am thinking of adding another strategy performance metric to > optimizer, to complement Net Profit, Max DD, Profit Factor, Kelly, and > PI. The new metric name would be "robustness", or its opposite, > "sensitivity". There is already a way to determine how sensitive the > strategy performance is with respect to changes in parameters, which > is the optimization maps, but when the number of parameters exceeds 2, > it becomes somewhat difficult to visualize it. The "robustness" metric > would be a single number which would measure the volatility of > performance results when the optimal strategy parameters change by a > certain small amount, say 10%. > > For example, consider these two strategies: > Strategy A has 3 parameters, 10-100-50. Its best factor is 3.0. When > its parameters change by 10% (such as 9-90-45), profit factor becomes > as low as 1.0 > Strategy B has 3 parameters, 200-10-10. Its profit factor is 3.0. When > its parameters change by 10% (such as 180-9-11), profit factor becomes > as low as 2.5 > > Clearly, strategy B is superior to strategy A because it is much less > sensitive to small changes in parameters. To put it in another way, > strategy B is more robust than strategy A. > > To calculate the "robustness" metric, we need a piece of code to > iterate through the brute force optimization results, and to compute > some sort of performance volatility measure. It could be standard > deviation of performance near a given set of parameters, or it could > simply be the "worst" performance near a given set of parameters. Once > calculated, the robustness metric would be shown as a column in the > optimization results table. > > At the moment, I am working on various other enhancements to JBT, so I > am looking for someone else to implement the robustness/sensitivity > metric. If you would like to code it, please respond in this thread. > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en -~----------~----~----~----~------~----~------~--~---
