Hi Richard -- Two books are already available. www.blueowlpress.com The third, Advanced AmiBroker, should be out by about April 2010 and will cover this topic. If I get it right, it should be as valuable to systems developers as the Vince and Tharp books, with a lot more practical information and implementation code.
Thanks, Howard On Thu, Oct 15, 2009 at 6:08 AM, Richard N Park <[email protected]>wrote: > > > Bravo, Howard! > > > > You should write a book. J > > > > > ------------------------------ > > *From:* [email protected] [mailto:[email protected]] *On > Behalf Of *Howard B > *Sent:* Thursday, October 15, 2009 7:46 AM > *To:* [email protected] > *Subject:* Re: [amibroker] Re: Is the Walk forward study useful? > > > > > > Hi Bing, and all -- > > > I think we need a reality check. > > First -- computing the t-test, or any other metric using the results from > in-sample runs has no value. Almost any trading system can have the > parameters, logic, time frame, and reporting period adjusted so that the > in-sample results are very profitable. In my speeches, I show a series of > slides, each slide displayed in two steps. Step 1 is the in-sample equity > after optimizing over an in-sample period. The chart ends at the end of the > in-sample period. It always looks good. When the results of testing a > system do not look good, we do not spend any more time on that system, but > go on to others where it does look good. Step 2 adds an out-of-sample data > period, applies the same system, and shows the out-of-sample equity. For > some systems the equity continues to rise, for some it goes flat, for others > it drops sharply. There is no way to tell what will happen in the > out-of-sample period without testing the out-of-sample period. That is -- > there is no information in the in-sample results that predict the > out-of-sample results. There is no way to know whether the model has fit > itself to the signal component of the data that contains the pattern leading > to profitable trades or to some noise component of the data that does not > exist in the out-of-sample data. > > Second -- if someone has a trading system that has produced a truly > out-of-sample set of closed trades where the t-test, or any other fitness > metric, would be embarrassingly high without limiting the value used as the > number of data points, he or she should call me. I can help them find a > semitrailer large enough to carry all the money they will make trading that > system. Limiting the number used as N is a non-issue, in addition to being > bad procedure. > > Tharp admits that he is not a statistician, and that he finds some of the > mathematics involved in position sizing and fitness function analysis to be > at his comfort limit. It is apparent to me that the systems he shows as > standards for SQNs of 2, 3, 4, 5, and so forth are not the result of trading > system runs -- either in-sample or out-of-sample -- to which he has applied > his metric; they are artificial examples constructed so the results turn out > as he wants them to. That is not a bad thing in itself, but when he > suggests that we should go on searching for systems with SQNs of 10 or more > (page 279 of "Definitive Guide to Position Sizing"), that is completely > unrealistic and will send naive systems developers off on windmill-tilting > quests that will never be successful. > > One of the difficulties using Tharp's data sets is that the standard > deviation of losing trades is zero for some of them. That is not only > unrealistic, but it makes computation of metrics that include standard > deviation of losing trades, such as Sortino ratio, difficult. > > Do the following experiment. Put together a data set that represents > potentially realistic trading results that you hope to achieve -- be > optimistic, but realistic. If you would expect one trade a week, a set of > 52 data points represents a year. 252 data points if you would have a > trading result every day. Each data point is the number of dollars gained > or lost from that closed trade, based on trading a single unit -- one > futures contract, one hundred shares of stock, or whatever your one unit > is. Compute the mean, standard deviation, and t-test score -- where the > t-test is "what is the level of confidence that the mean is greater than > 0". Put the data into a text file and use it as input to Equity Monaco, > ProSizer, or Market System Analyzer. Run some Monte Carlo simulations to > see what the equity could look like from trading one year. Systems with > t-test scores, based on expectancy, of 3 will make you incredible amounts of > money. Dummy up a data set with a t-test / SQN of 10 and you will see how > unrealistic that is. > > The critical elements needed to apply aggressive position sizing are: > trading something that can be scaled up as desired, positive expectancy, > limited semi-deviation (standard deviation of losing trades), and frequent > trading. > > Being able to limit the amount lost on a losing trade is essential. Risk > of bankruptcy (and all trading systems have a non-zero probability of going > bankrupt) increases dramatically as the standard deviation of losing trades > increases. Dummy up some data and runs some more tests. The effect the > amount lost on losing trades has on system performance is scary. > > Now go back to your trading system development platform (AmiBroker, of > course) and test your trading systems, this time focusing on limiting the > standard deviation of losing trades. If your system holds a long time (more > than a week or two), pay attention to Maximum Adverse Excursion. If you > attempt to apply aggressive position sizing to a system that has reasonable > results for closed trades, but has large MAE intra-trade, you will get > stopped out (or scared out) intra-trade. > > Remember -- if you are applying position sizing to your trading system, > your largest loss will come when you have your largest position. Read Ralph > Vince. > > As usual -- be careful to base your analysis and decision whether to trade > any system on truly out-of-sample results. Decisions based on in-sample or > contaminated out-of-sample results seriously underestimate the probability > of bankruptcy. > > Repeat after me -- keep a positive expectancy, limit losing trades, trade > frequently. > > Thanks for listening, > Howard > > On Thu, Oct 15, 2009 at 1:11 AM, bingk66 <[email protected]> > wrote: > > > > Hi Howard, > > If there are no means to limit the number of transactions in the calcs, > then one seriously runs the risk of challenging the mystical t-test score of > 7 that you spoke about previously. > > As an example, if the OOS test was run over a 5 year period with 5000 > transactions (a mere 1000 transaction/year, which is not excessive, > especially for very short term trades), sqrt(5000) alone would yield in > excess of 70 for the multiplier. This would leave expectancy/StdDev of R > with just a target of 0.1, to reach the 7 t-tests score. > > Now, if you had 1,000,000 tranasctions in your OOS test.... > > The concept of limiting the trade count does make sense to me. Maybe 100 is > too low, and should be set higher. There does come a point whereby the > sqrt(N) part of the equation will render the rest of the equation irrelevant > once N gets too large. > > $0.02 > > Bing > > > > --- In [email protected] <amibroker%40yahoogroups.com>, Howard B > <howardba...@...> wrote: > > > > > Hi Zozu -- > > > > I must disagree with Van Tharp on this. > > > > If the runs are truly out-of-sample, then each and every one contributes > to > > the computation. It makes no sense to limit the count to 100. It is poor > > procedure to limit the count. It is bad science to limit the count. Do > not > > limit the count. > > > > If the runs are in-sample, then the test has no meaning anyway. Computing > > the t-test statistic using any N will be misleading. Do not even do the > > computation. If a decision to trade a system is made after computing the > > t-test statistic on trades that came solely from in-sample results, there > is > > an extremely high probability that a Type I error will be committed. That > > is, the trader will believe that his system is better than random, when > it > > is in fact not better than random. Type I errors result in loss of money. > > > > Thanks, > > Howard > > > > > > > On Tue, Oct 13, 2009 at 10:54 AM, zozuzoza <zoz...@...> wrote: > > > > > > > > > > > Hi Howard, > > > > > > Limiting the number of N doesn't mean that you are not using all trades > for > > > the calculation of SQN. Only the sqrt(N) part of the formula is limited > in > > > order not to distort the results if there are many trades. It makes > sense. > > > The other part of the formula does count on all the trades. > > > > > > Zozu > > > > > > > > > > > > > > >
