Howard, Thank you for your patience and detailed reply.
I won't comment further until I have read your book. No it doesn't seem hard. Regards, BrianB2. --- In [email protected], "Howard B" <[EMAIL PROTECTED]> wrote: > > Hi Brian -- > > Thanks for your comments. First my response regarding objective functions. > > The choice of the objective function is strictly a management decision, and > must be made before any serious testing begins. All sub-objectives must be > combined into The single-valued objective function. The programmer has > nothing to do with the choice of the objective function and the objective > function is not available for optimization. If the manager happens to also > do the programming, then it is with the manager's hat on that the objective > function is chosen. > > The Quantitative Trading Systems book has a chapter on objective functions, > and a lot of material throughout the book discussing the application of > objective functions to testing and optimization. There is an appendix that > describes in complete detail how to combine sub-objectives to create a > single-valued objective function, and then use that objective function in > AmiBroker testing and optimization. Whether there is only one metric that > is important to the trader / manager or 20, they must each be evaluated in > terms of importance, have a range of acceptable values determined, assigned > a relative weight, and combined. > > Think ahead to the automated walk-forward procedure we all must use to > validate a trading system. At the end of the search / optimization for each > in-sample period, the Best set of values must be chosen, and these values > used to evaluate the out-of-sample period. At the end of the last available > in-sample period, the concatenated results from all the out-of- sample > periods are evaluated. If those results are acceptable, then the Best set > of values from the final in-sample optimization will be used to trade the > system with real money. There can be only one Best. If the manager has an > opportunity to look at the trades associated with each of the alternatives > to the best, and if that manager picks any set of trades -- that is, trades > associated with a set of values -- other than the one that is ranked > highest, then that manager is saying that there are additional metrics that > are not in the current objective function, but that must be added to it. > The set of values that are ranked highest Must represent the trading that > the manager believes is Best for himself, herself, or the organization. > > Switching to questions about in-sample testing and out-of-sample testing. > > The Only results that matter are those observed in Strictly out-of- sample > testing. There is No number of in-sample trades that are adequate to > validate a trading system. Not 30, not 130! I have documented two separate > trading systems that each have over one million (sic) closed trades > in-sample and appear to be profitable, yet neither of those systems > validates out-of-sample. > > In-sample results are Meaningless!!!! They are always good. We do not > quite searching / optimizing until they are good. Basing a decision to > trade based on in-sample results is dangerous to the wealth. > > Remember -- truly out-of-sample testing begins tomorrow with real money. > Only rigorous validation procedures can prepare us for that. And even if a > trading system seems to validate, there is no guarantee of future > profitability -- just an increased probability of future profitability. > > I realize these statements sound hard -- both in tone and in difficulty of > implementation. It is Absolutely Necessary that every systems developer who > hopes to be profitable understand and implement objective functions and > in-sample / out-of-sample testing. Unfortunately, there is no easy > solution. Until Tomasz provided the fantastic tools within AmiBroker, there > was not even a difficult solution. > > Thanks for listening, > Howard > www.quantitativetradingsystems.com > > > > > On 3/17/07, brian.z123 <[EMAIL PROTECTED]> wrote: > > > > Howard, > > > > Thanks for your post. > > I value the opportunity to talk to quality people on a quality topic. > > > > In laymans terms; does the servitude to the objective function > > require compromising the multiple sub-objectives? > > If so who makes that decision; when, where and how is it made? > > According to the Calresco site, these decisions are made by the > > *programmer* on a subjective basis. > > > > If there are 3 or more sub-objectives with, with 2 or 3 metrics for > > each and, say a range of 20 for each of the metrics parameters , > > doesn't the range of possible compromises become somewhat wide? > > > > If we could extract the metrics for each of those multiple paths on a > > point by point basis, I wonder if, with hindsight, *we* might like to > > change our prior subjective choices. > > > > Re walk OOS/walk-forward. > > > > A money game derived from the toss of a fair coin with +1 and -1 > > win/loss values assigned to each side of the coin is played by 1000 > > people. > > They are not aware of the nature of the game, only their individual > > outcomes i.e. their equity curves or trade value time series as they > > unfold. > > > > An observer, who is a mathematician, knows the game is break-even and > > also all of the metrics for the game in advance, but the players > > don't. > > > > At the end of 1000 plays, in all probability, not one single player > > will have exactly broken even. > > Approx 60% of players will have a small win or loss record, while on > > the other end of the scale < 5% of players will have either an > > extreme win or extreme loss outcome. > > In all probability not one single equity curve will be exactly the > > same as any other. > > > > If one of the extreme winners happened to be a mathematician and > > decided to calculate his/her future in the game, would muliple valued > > objective analysis, arrive at any conclusion other than that it is a > > rosy one? > > By any measure, would the extreme winners not be justified in the > > belief that their future in the game is rosy? > > > > To the observer the 1000 equity curves, when expressed as a > > probability, represent a good approximation of all possible future > > outcomes of playing the game, for that exact number of plays. > > This is true to the basic tenents of maths, which state that the > > future can only be described in terms of probability. > > > > If all of the equity outcomes are written on a marble, placed in a > > basket, and drawn at random, this test would represent a true model > > of a blind or walk forward test. > > > > For a trader, who has backtested, the basket also represents > > historical results. > > > > Go ahead and draw your 2 marbles; one for your in sample (IS) equity > > curve and one for your out of sample (OOS) equity curve. > > > > What is the chance that your first outcome will truely represent your > > future in the game? > > > > What is the chance that your second equity curve will truely > > represent your future in the game? > > > > What is the chance that your first equity curve will be within coeee > > of (Aussie slang for close to) the first? > > Should we define close as 50%, as Mr Pardo does? > > Why not 49 or 51? > > > > Does the first marble tell you (more/less/the same) about your future > > in the game, compared to the second? > > > > What are the chances that your first and second marbles will both be > > *good* ones? > > Is that a better or worse *sign* than drawing one *good* one followed > > by one *bad* one? > > > > BrianB2 *:-) > > > > > > --- In [email protected] <amibroker% 40yahoogroups.com>, "Howard B" > > <howardbandy@> wrote: > > > > > > Greetings -- > > > > > > I'd like to add a comment on multivalued objective functions, > > particularly > > > as they relate to Monte Carlo analysis and walk-forward testing. > > > > > > As your trading system development moves to the stage of having a > > > walk-forward process performed automatically, it needs to be guided > > by an > > > objective function that incorporates all of the features that are > > important > > > to you and can be expressed as a single scalar value. > > > > > > As you know, the walk-forward process divides the entire time > > series being > > > used into a sequence of in-sample periods, each followed by an out- > > of-sample > > > period. A search procedure picks the single Best set of values > > for the > > > trading system's optimizable variables for a given in-sample > > period, then > > > records the results of using those values to simulate trading over > > the > > > out-of-sample period. Then it slides the starting dates for both > > the > > > in-sample period and out-of-sample period forward, usually by the > > length of > > > the out-of-sample period, and does the search over again. This > > process > > > continues until all of the data has been processed. The results > > from the > > > out-of-sample periods are concatenated together and are used to > > decide > > > whether the trading system is a good one or not. > > > > > > The key point here is this: the search procedure must make its > > decision on > > > which set of variables is Best based on a single value -- the value > > of the > > > objective function. By the time the development reaches this > > stage, we, as > > > system developers or programmers, will not have an opportunity to > > look down > > > the list of alternative trading systems to see if we would have > > picked one > > > other than the one at the top of the list. So, as we are working > > with > > > multivalued objective functions, we must incorporate them into a > > > single-valued objective function that fits our trading requirements > > or > > > personality and that we trust to sort the alternatives into the > > order we > > > prefer. > > > > > > AmiBroker has the capability to creating custom objective > > functions. There > > > is an extensive discussion in my book about objective functions. > > The > > > discussion includes an example of using AmiBroker's custom > > backtester to > > > create a single-valued objective function by starting with a central > > > objective function, then applying penalties (which can be positive > > or > > > negative) to it to take secondary goals into account. > > > > > > In fact, I think that objective functions are so important that > > selection of > > > the objective function should be the First step in trading system > > design. > > > If the objective function fits the trader, most of the problems > > related to > > > the difficulty of following a trading system and of the psychology > > of > > > trading disappear. > > > > > > Thanks for listening, > > > Howard > > > www.quantitativetradingsystems.com > > > > > > > > > > > > > > > > > > > > > On 17 Mar 2007 03:06:02 -0700, thomasdrewyallop <drewyallop@> > > wrote: > > > > > > > > Hello all, > > > > > > > > I have been working on the MCP technique described in Aronson's > > book > > > > for some time now. I have just completed conversion of the C++ > > code on > > > > the web site to C# plus some associated utlities to massage AB > > data > > > > into the required format yet. No test yet; I will update under > > this > > > > thread. > > > > > > > > A few words on the theoretical underpinnings. There has been new > > > > information since the book was published and the code written. I > > > > believe an update is in the works. Also you need to be cautious > > when > > > > running MCP on IS data. This is only valid under certain > > conditions. > > > > Otherwise you must run OOS. I had an email from Aronson > > explaining all > > > > this but can't find it. You might want to contact David directly - > > a > > > > good guy and willing to talk with readers. > > > > > > > > Finally, I would not reject walk forward. A very useful technique > > > > despite Aronson's reservations. Well integrated with AB too via > > Fred > > > > Tonetti's IO add-in. > > > > > > > > Best regards, > > > > > > > > Drew Yallop > > > > > > > > p.s. just remebered that there is discussion on MCP as a possible > > > > future addition to AB. Look in the AB suggestions section of the > > web site. > > > > > > > > > > > > > > > > > > > > > >
