Hi Brian -- Thanks for your comments. First my response regarding objective functions.
The choice of the objective function is strictly a management decision, and must be made before any serious testing begins. All sub-objectives must be combined into The single-valued objective function. The programmer has nothing to do with the choice of the objective function and the objective function is not available for optimization. If the manager happens to also do the programming, then it is with the manager's hat on that the objective function is chosen. The Quantitative Trading Systems book has a chapter on objective functions, and a lot of material throughout the book discussing the application of objective functions to testing and optimization. There is an appendix that describes in complete detail how to combine sub-objectives to create a single-valued objective function, and then use that objective function in AmiBroker testing and optimization. Whether there is only one metric that is important to the trader / manager or 20, they must each be evaluated in terms of importance, have a range of acceptable values determined, assigned a relative weight, and combined. Think ahead to the automated walk-forward procedure we all must use to validate a trading system. At the end of the search / optimization for each in-sample period, the Best set of values must be chosen, and these values used to evaluate the out-of-sample period. At the end of the last available in-sample period, the concatenated results from all the out-of-sample periods are evaluated. If those results are acceptable, then the Best set of values from the final in-sample optimization will be used to trade the system with real money. There can be only one Best. If the manager has an opportunity to look at the trades associated with each of the alternatives to the best, and if that manager picks any set of trades -- that is, trades associated with a set of values -- other than the one that is ranked highest, then that manager is saying that there are additional metrics that are not in the current objective function, but that must be added to it. The set of values that are ranked highest Must represent the trading that the manager believes is Best for himself, herself, or the organization. Switching to questions about in-sample testing and out-of-sample testing. The Only results that matter are those observed in Strictly out-of-sample testing. There is No number of in-sample trades that are adequate to validate a trading system. Not 30, not 130! I have documented two separate trading systems that each have over one million (sic) closed trades in-sample and appear to be profitable, yet neither of those systems validates out-of-sample. In-sample results are Meaningless!!!! They are always good. We do not quite searching / optimizing until they are good. Basing a decision to trade based on in-sample results is dangerous to the wealth. Remember -- truly out-of-sample testing begins tomorrow with real money. Only rigorous validation procedures can prepare us for that. And even if a trading system seems to validate, there is no guarantee of future profitability -- just an increased probability of future profitability. I realize these statements sound hard -- both in tone and in difficulty of implementation. It is Absolutely Necessary that every systems developer who hopes to be profitable understand and implement objective functions and in-sample / out-of-sample testing. Unfortunately, there is no easy solution. Until Tomasz provided the fantastic tools within AmiBroker, there was not even a difficult solution. Thanks for listening, Howard www.quantitativetradingsystems.com On 3/17/07, brian.z123 <[EMAIL PROTECTED]> wrote:
Howard, Thanks for your post. I value the opportunity to talk to quality people on a quality topic. In laymans terms; does the servitude to the objective function require compromising the multiple sub-objectives? If so who makes that decision; when, where and how is it made? According to the Calresco site, these decisions are made by the *programmer* on a subjective basis. If there are 3 or more sub-objectives with, with 2 or 3 metrics for each and, say a range of 20 for each of the metrics parameters , doesn't the range of possible compromises become somewhat wide? If we could extract the metrics for each of those multiple paths on a point by point basis, I wonder if, with hindsight, *we* might like to change our prior subjective choices. Re walk OOS/walk-forward. A money game derived from the toss of a fair coin with +1 and -1 win/loss values assigned to each side of the coin is played by 1000 people. They are not aware of the nature of the game, only their individual outcomes i.e. their equity curves or trade value time series as they unfold. An observer, who is a mathematician, knows the game is break-even and also all of the metrics for the game in advance, but the players don't. At the end of 1000 plays, in all probability, not one single player will have exactly broken even. Approx 60% of players will have a small win or loss record, while on the other end of the scale < 5% of players will have either an extreme win or extreme loss outcome. In all probability not one single equity curve will be exactly the same as any other. If one of the extreme winners happened to be a mathematician and decided to calculate his/her future in the game, would muliple valued objective analysis, arrive at any conclusion other than that it is a rosy one? By any measure, would the extreme winners not be justified in the belief that their future in the game is rosy? To the observer the 1000 equity curves, when expressed as a probability, represent a good approximation of all possible future outcomes of playing the game, for that exact number of plays. This is true to the basic tenents of maths, which state that the future can only be described in terms of probability. If all of the equity outcomes are written on a marble, placed in a basket, and drawn at random, this test would represent a true model of a blind or walk forward test. For a trader, who has backtested, the basket also represents historical results. Go ahead and draw your 2 marbles; one for your in sample (IS) equity curve and one for your out of sample (OOS) equity curve. What is the chance that your first outcome will truely represent your future in the game? What is the chance that your second equity curve will truely represent your future in the game? What is the chance that your first equity curve will be within coeee of (Aussie slang for close to) the first? Should we define close as 50%, as Mr Pardo does? Why not 49 or 51? Does the first marble tell you (more/less/the same) about your future in the game, compared to the second? What are the chances that your first and second marbles will both be *good* ones? Is that a better or worse *sign* than drawing one *good* one followed by one *bad* one? BrianB2 *:-) --- In [email protected] <amibroker%40yahoogroups.com>, "Howard B" <[EMAIL PROTECTED]> wrote: > > Greetings -- > > I'd like to add a comment on multivalued objective functions, particularly > as they relate to Monte Carlo analysis and walk-forward testing. > > As your trading system development moves to the stage of having a > walk-forward process performed automatically, it needs to be guided by an > objective function that incorporates all of the features that are important > to you and can be expressed as a single scalar value. > > As you know, the walk-forward process divides the entire time series being > used into a sequence of in-sample periods, each followed by an out- of-sample > period. A search procedure picks the single Best set of values for the > trading system's optimizable variables for a given in-sample period, then > records the results of using those values to simulate trading over the > out-of-sample period. Then it slides the starting dates for both the > in-sample period and out-of-sample period forward, usually by the length of > the out-of-sample period, and does the search over again. This process > continues until all of the data has been processed. The results from the > out-of-sample periods are concatenated together and are used to decide > whether the trading system is a good one or not. > > The key point here is this: the search procedure must make its decision on > which set of variables is Best based on a single value -- the value of the > objective function. By the time the development reaches this stage, we, as > system developers or programmers, will not have an opportunity to look down > the list of alternative trading systems to see if we would have picked one > other than the one at the top of the list. So, as we are working with > multivalued objective functions, we must incorporate them into a > single-valued objective function that fits our trading requirements or > personality and that we trust to sort the alternatives into the order we > prefer. > > AmiBroker has the capability to creating custom objective functions. There > is an extensive discussion in my book about objective functions. The > discussion includes an example of using AmiBroker's custom backtester to > create a single-valued objective function by starting with a central > objective function, then applying penalties (which can be positive or > negative) to it to take secondary goals into account. > > In fact, I think that objective functions are so important that selection of > the objective function should be the First step in trading system design. > If the objective function fits the trader, most of the problems related to > the difficulty of following a trading system and of the psychology of > trading disappear. > > Thanks for listening, > Howard > www.quantitativetradingsystems.com > > > > > > > On 17 Mar 2007 03:06:02 -0700, thomasdrewyallop <[EMAIL PROTECTED]> wrote: > > > > Hello all, > > > > I have been working on the MCP technique described in Aronson's book > > for some time now. I have just completed conversion of the C++ code on > > the web site to C# plus some associated utlities to massage AB data > > into the required format yet. No test yet; I will update under this > > thread. > > > > A few words on the theoretical underpinnings. There has been new > > information since the book was published and the code written. I > > believe an update is in the works. Also you need to be cautious when > > running MCP on IS data. This is only valid under certain conditions. > > Otherwise you must run OOS. I had an email from Aronson explaining all > > this but can't find it. You might want to contact David directly - a > > good guy and willing to talk with readers. > > > > Finally, I would not reject walk forward. A very useful technique > > despite Aronson's reservations. Well integrated with AB too via Fred > > Tonetti's IO add-in. > > > > Best regards, > > > > Drew Yallop > > > > p.s. just remebered that there is discussion on MCP as a possible > > future addition to AB. Look in the AB suggestions section of the web site. > > > > > > >
