Howard, Thanks for your post. I value the opportunity to talk to quality people on a quality topic.
In laymans terms; does the servitude to the objective function require compromising the multiple sub-objectives? If so who makes that decision; when, where and how is it made? According to the Calresco site, these decisions are made by the *programmer* on a subjective basis. If there are 3 or more sub-objectives with, with 2 or 3 metrics for each and, say a range of 20 for each of the metrics parameters , doesn't the range of possible compromises become somewhat wide? If we could extract the metrics for each of those multiple paths on a point by point basis, I wonder if, with hindsight, *we* might like to change our prior subjective choices. Re walk OOS/walk-forward. A money game derived from the toss of a fair coin with +1 and -1 win/loss values assigned to each side of the coin is played by 1000 people. They are not aware of the nature of the game, only their individual outcomes i.e. their equity curves or trade value time series as they unfold. An observer, who is a mathematician, knows the game is break-even and also all of the metrics for the game in advance, but the players don't. At the end of 1000 plays, in all probability, not one single player will have exactly broken even. Approx 60% of players will have a small win or loss record, while on the other end of the scale < 5% of players will have either an extreme win or extreme loss outcome. In all probability not one single equity curve will be exactly the same as any other. If one of the extreme winners happened to be a mathematician and decided to calculate his/her future in the game, would muliple valued objective analysis, arrive at any conclusion other than that it is a rosy one? By any measure, would the extreme winners not be justified in the belief that their future in the game is rosy? To the observer the 1000 equity curves, when expressed as a probability, represent a good approximation of all possible future outcomes of playing the game, for that exact number of plays. This is true to the basic tenents of maths, which state that the future can only be described in terms of probability. If all of the equity outcomes are written on a marble, placed in a basket, and drawn at random, this test would represent a true model of a blind or walk forward test. For a trader, who has backtested, the basket also represents historical results. Go ahead and draw your 2 marbles; one for your in sample (IS) equity curve and one for your out of sample (OOS) equity curve. What is the chance that your first outcome will truely represent your future in the game? What is the chance that your second equity curve will truely represent your future in the game? What is the chance that your first equity curve will be within coeee of (Aussie slang for close to) the first? Should we define close as 50%, as Mr Pardo does? Why not 49 or 51? Does the first marble tell you (more/less/the same) about your future in the game, compared to the second? What are the chances that your first and second marbles will both be *good* ones? Is that a better or worse *sign* than drawing one *good* one followed by one *bad* one? BrianB2 *:-) --- In [email protected], "Howard B" <[EMAIL PROTECTED]> wrote: > > Greetings -- > > I'd like to add a comment on multivalued objective functions, particularly > as they relate to Monte Carlo analysis and walk-forward testing. > > As your trading system development moves to the stage of having a > walk-forward process performed automatically, it needs to be guided by an > objective function that incorporates all of the features that are important > to you and can be expressed as a single scalar value. > > As you know, the walk-forward process divides the entire time series being > used into a sequence of in-sample periods, each followed by an out- of-sample > period. A search procedure picks the single Best set of values for the > trading system's optimizable variables for a given in-sample period, then > records the results of using those values to simulate trading over the > out-of-sample period. Then it slides the starting dates for both the > in-sample period and out-of-sample period forward, usually by the length of > the out-of-sample period, and does the search over again. This process > continues until all of the data has been processed. The results from the > out-of-sample periods are concatenated together and are used to decide > whether the trading system is a good one or not. > > The key point here is this: the search procedure must make its decision on > which set of variables is Best based on a single value -- the value of the > objective function. By the time the development reaches this stage, we, as > system developers or programmers, will not have an opportunity to look down > the list of alternative trading systems to see if we would have picked one > other than the one at the top of the list. So, as we are working with > multivalued objective functions, we must incorporate them into a > single-valued objective function that fits our trading requirements or > personality and that we trust to sort the alternatives into the order we > prefer. > > AmiBroker has the capability to creating custom objective functions. There > is an extensive discussion in my book about objective functions. The > discussion includes an example of using AmiBroker's custom backtester to > create a single-valued objective function by starting with a central > objective function, then applying penalties (which can be positive or > negative) to it to take secondary goals into account. > > In fact, I think that objective functions are so important that selection of > the objective function should be the First step in trading system design. > If the objective function fits the trader, most of the problems related to > the difficulty of following a trading system and of the psychology of > trading disappear. > > Thanks for listening, > Howard > www.quantitativetradingsystems.com > > > > > > > On 17 Mar 2007 03:06:02 -0700, thomasdrewyallop <[EMAIL PROTECTED]> wrote: > > > > Hello all, > > > > I have been working on the MCP technique described in Aronson's book > > for some time now. I have just completed conversion of the C++ code on > > the web site to C# plus some associated utlities to massage AB data > > into the required format yet. No test yet; I will update under this > > thread. > > > > A few words on the theoretical underpinnings. There has been new > > information since the book was published and the code written. I > > believe an update is in the works. Also you need to be cautious when > > running MCP on IS data. This is only valid under certain conditions. > > Otherwise you must run OOS. I had an email from Aronson explaining all > > this but can't find it. You might want to contact David directly - a > > good guy and willing to talk with readers. > > > > Finally, I would not reject walk forward. A very useful technique > > despite Aronson's reservations. Well integrated with AB too via Fred > > Tonetti's IO add-in. > > > > Best regards, > > > > Drew Yallop > > > > p.s. just remebered that there is discussion on MCP as a possible > > future addition to AB. Look in the AB suggestions section of the web site. > > > > > > >
