Mike, > Along these same lines, I recently posted comments (including code) > for a similar approach
I somehow missed your earlier comments (I usually grab the stats stuff). Thanks for helping us along and pointing to your previous work. You're stretching me there. brian_z --- In [email protected], "Mike" <[EMAIL PROTECTED]> wrote: > > Hi, > > The quickest way to do it might be to run your backtest, output the > trade list to .csv file for loading into Excel, then do the same > for a Random strategy. You could then use Excel to perform a t-test > (equivalent to z-test after sufficient data points) as follows: > > - Add the "Analysis TookPak" to Excel via "Tools | Add ins..." menu. > - Run "Tools | Data Analysis..." menu > - Select "t-Test: Two variables assuming equal variances". > - Select the trade percentage return column from each of the strategy > results for the variable ranges. > - Enter zero as the hypothesized mean difference > - Indicate where you want the resulting table to get placed > - Click OK > > In the resulting table, if you see P(T<=t) one tail with a value > of .05 or less, then you've got statistical significance with 95% > confidence. > > Along these same lines, I recently posted comments (including code) > for a similar approach based on applying your strategy signals > against detrended daily market change data (David Aronson's book). > The end result is the same; comparing your strategy against random > returns. > > The first script (message 120676) will detrend the average daily > change of the market such that the average change will be zero. This > implies that if we were to randomly apply signals for long and short > against the data, the return would be zero. > > The second script (message 120718) gives an example "always in" > strategy that applies its market signals against the detrended daily > changes. > > You would then run the t-test between the means of the two sets of > results (i.e the zero mean of the detrended daily change against the > presumably non zero mean of your strategy - as applied against the > detrended daily change). > > To prepare the results for export to Excel, add the following lines > to the end of the first script (as opposed to having to cut and paste > from the Quote Editor of the resulting composite): > > SetForeign("~" + Name()); > Filter = NOT IsNan(Close); > AddColumn(Close, "Detrended Ln", 10.9); > RestorePriceArrays(); > > Similarly, add these lines to the bottom of the second script: > > SetForeign("~" + Name() + "Return"); > Filter = NOT IsNan(Close); > AddColumn(Close, "Detrended Result", 10.9); > RestorePriceArrays(); > > Now after running the first script as an exploration against your > symbol, you can just export the result to .csv and load into Excel. > > Same for running the second script as an exploration against that > same symbol. Then perform t-test as above using the two output > columns as the variable ranges. > > Note that the values are logarithms of the percent daily change, so > don't try interpreting them as price values. > > Mike > > > --- In [email protected], Thomas Ludwig <Thomas.Ludwig@> > wrote: > > > > Hello, > > > > in "Quantitative Trading Systems" on p. 256, Howard describes a z- > score > > test in order to evaluate the statistical significance of a trading > > system. While the formula is easy to write in AFL, I don't think > that > > it can be done as a custom metric since the system to be evaluated > is > > compared with a Random System. Any idea how to sensibly implement > it in > > Amibroker? > > > > I'm using another statistical test proposed by the late Arthur > Merrill > > some years ago in S&C. It's the "chi squared with one degree of > > freedom, with the Yates correction". Here's how I implemented it in > AB: > > > > //chi squared with one degree of freedom, with the Yates correction > > wi=st.GetValue("WinnersQty"); > > Lo=st.GetValue("LosersQty"); > > Chi = (abs(wi-Lo)-1)^2/(wi+Lo); > > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very > > significant(1000:1), >6.64: significant (100:1) , >3.84: probably > > significant (20:1), <3.84: significance doubtful", Chi ); > > > > While this metric doesn't tell you anything if your system is > > profitable, it tells you if its signals are only pure coincidence > > (simply put). It's remarkable that many systems that seem to be > > promising according to the usual metrics, are below 3.84, i.e. > > significance doubtful. You need either a rather high number of > trades > > or a very high percentage of winning trades to shift this metric > > significantly higher. At least for (medium-term) EOD systems > (that's > > what I trade) this is not easy to achieve. > > > > What do you think about this metric? Are there other "better" > > statistical metrics? If yes - would you mind sharing the AFL code? > > > > Best regards, > > > > Thomas > > >
