> Some of the 'academic' stats methods are not quite as good a fit to > trading as we first assume e.g. look at the recent discussion on > Fourier Transform, which comes from Signal Processing (electronics?) > and falls down rather spectacularly in trading because stockmarket > data isn't stationary.
For what is worth, FFT is most often applied to audio signals, that are definitelly NOT stationary :-) so that argument is not really valid :-) There are methods (windowing+padding) to increase FFT resolution on short data samples, and there are also methods different than FFT for spectal analysis. But that is rather broad subject .... Best regards, Tomasz Janeczko amibroker.com ----- Original Message ----- From: "brian_z111" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Wednesday, March 12, 2008 12:41 AM Subject: [amibroker] Re: Statistical tests as custom metrics > Thomas, > > I have included Jeffs question with your opening query since they fit > so closely. > >> looking at your algorithm I'm not seeing expected. What exactly >>are >> you test significance against? > > The expected value of an unbiased binomial event is 0.50 (50% win). > > Since the (equity) market has a slight upward bias we need to use an > expected value, that includes the market bias, as the base line (or > detrend as per Howards/Aronsons methods). > > (Personally I don't detrend as I like to condition myself to see the > markets as they actually are, and mentally factor in the trend, so I > use market bias benchmarking). > >> > in "Quantitative Trading Systems" on p. 256, Howard describes a z- >> score >> > test in order to evaluate the statistical significance of a > trading >> > system. While the formula is easy to write in AFL, I don't think >> that >> > it can be done as a custom metric since the system to be > evaluated >> is >> > compared with a Random System. Any idea how to sensibly implement >> it in >> > Amibroker? > > At page 91,in his book, (Entries and Exits chapter) Howard gives some > very good (random entry) examples of how we can get an estimation for > the 'standardized binomial expectancy' of any market i.e. you can get > the mean expected wins for the actual market you are testing your > system in and use that in the Z score calculation - I think you would > be better to use random entries with an exit after a set number of > days == the average time your system trades are in the market. > > I am still learning AB myself so I am not sure if we can implement > Howards Z equation directly in AB - you will probably have to do it > outside somewhere - I haven't figured out how we can get the SD of a > trade series (from a backtest) in AB - anyway we don't have built in > stats tables (I suppose you could manually plug in the typical Z > scores). > > I'm exporting to Excel and doing my evaluations there, but I don't > get that fancy. > > > > I'm using another statistical test proposed by the late Arthur >> Merrill >> > some years ago in S&C. It's the "chi squared with one degree of >> > freedom, with the Yates correction". Here's how I implemented it > in >> AB: >> > >> > //chi squared with one degree of freedom, with the Yates > correction >> > wi=st.GetValue("WinnersQty"); >> > Lo=st.GetValue("LosersQty"); >> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo); >> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very >> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably >> > significant (20:1), <3.84: significance doubtful", Chi ); >> > >> > What do you think about this metric? > > I think it is a very conservative measure. > > One of the problems we have, in evaluation, is that 'academic' > statistics filtered into freelance trading via institutional > investing - nothing against academics or institutional traders but > their focus is somewhat different to freelance traders. > > Some of the 'academic' stats methods are not quite as good a fit to > trading as we first assume e.g. look at the recent discussion on > Fourier Transform, which comes from Signal Processing (electronics?) > and falls down rather spectacularly in trading because stockmarket > data isn't stationary. > > Most of the stats we are using assume stationarity and also assume > that data will be normal/ random i.e. it will have a normal > distribution and that the datapoints are independent of each other. > > Neither is absolutely true, so the stats we are using are > approximations (of course the data we are using is only an > approximation anyway) - hence the doubts about Merrill's Chi. > >> > While this metric doesn't tell you anything if your system is >> > profitable, it tells you if its signals are only pure coincidence >> > (simply put). It's remarkable that many systems that seem to be >> > promising according to the usual metrics, are below 3.84, i.e. >> > significance doubtful. You need either a rather high number of >> trades >> > or a very high percentage of winning trades to shift this metric >> > significantly higher. At least for (medium-term) EOD systems >> (that's >> > what I trade) this is not easy to achieve. >> > > > Yes, it is very hard to find good trading systems. > > This is what I have found - many tests that come up with nothing, > especially in the first two years. > >>>Are there other "better" >> > statistical metrics? If yes - would you mind sharing the AFL code? >> > > > Try Howards Z method, using his random code, to find your expected > win rate for your market and see how that works out. > > I have started some original (to me) work, based on binomial > simulation of equity curves and the behaviour of random, 50/50, > trading systems. > > It is only at the experimental, concept stage. > I intend posting it to the UKB one day so that the mathematically > trained people in the forum can critique it (it might be a load of > old rubbish for all I know). > > Based on that work I am using PowerFactor, with sample error, to > guestimate significance (I can quickly do that in my head). > > Note that in PowerFactor the binomial component is considered to be > Gaussian, with independent variables, while the distribution of the > trades (ave%won/ave%lost) is not. > > Because of that I only apply the significance test to the W/L > binomial component (I claim that carrying out stats analysis on the > compound system results is biased because of the non-normal nature of > the distribution etc). > > For binomial events: > > variance == sample error (sort of) > > For 100 trades: > > sample error = +_10%; > expected random result (benchmark) == 50 wins; > > a no win trade will have a range of: > > 45-50 wins (one standard dev) > 40-60 wins (two standard devs) > 35- 65 (three standard devs) etc > > So for 100 trades 60 wins doesn't happen all that often, if the coin > is a fair coin (a random event). > > We are defintely going to take notice of 60/100 wins BUT it is > not 'out of this world' and we do not have certainty - we only have > the expectation that it is good - reality can, and does, dash > expectations on occasion. > > Because of this, W/L results are never a sure thing. > > To ensure against this take control of the ave%W/ave%L ratio - that > is something we can control via stops - if the W/L ratio turns out to > be a 'BlackSwan' our good stops will save us from crashing and > burning (keep us at low drawdowns). > > Comparing to Chi (for 100 trades with a 60% win record): > > Chi = (abs(wi-Lo)-1)^2/(wi+Lo); > > == ((60-40)-1)^2/(wi+Lo); > == 19^2/100; > == 361/100 > == 3.6 > == Not significant according to Chi but significant according to > brian (always look on the bright side of life!). > > I agree that finding 60% winners, in any market or timeframe, is very > difficult - that is the reality of trading. > > This problem is especially prevalent in mid - long term trading - say > indicators with long lookbacks are used - then the number of signals > available tends towards becoming a rare event and the trader then can > only see a small part of the longterm (10000 plus) trades - the > trader soon runs out of clean data and can't get high enough trade > counts. > > That is why I like shorter term trading (intraday to 2-3 day cycles) > where I can take advantage of statistical smoothing (I quickly > approach my theoretical edge i.e. relative to the calendar days). > > > As I said - please use 'my' theories at your own risk, at least until > after I post on the topic, and the mathematicians in the forum have a > chance to bash up my hypotheses. > > > brian_z > > > > > > > --- In [email protected], "jeffro861" <[EMAIL PROTECTED]> wrote: >> >> Ok, so the chi-squared tests for independence (real vs. expected) > so, >> looking at your algorithm I'm not seeing expected. What exactly > are >> you test significance against? >> >> --- In [email protected], Thomas Ludwig <Thomas.Ludwig@> >> wrote: >> > >> > Hello, >> > >> > in "Quantitative Trading Systems" on p. 256, Howard describes a z- >> score >> > test in order to evaluate the statistical significance of a > trading >> > system. While the formula is easy to write in AFL, I don't think >> that >> > it can be done as a custom metric since the system to be > evaluated >> is >> > compared with a Random System. Any idea how to sensibly implement >> it in >> > Amibroker? >> > >> > I'm using another statistical test proposed by the late Arthur >> Merrill >> > some years ago in S&C. It's the "chi squared with one degree of >> > freedom, with the Yates correction". Here's how I implemented it > in >> AB: >> > >> > //chi squared with one degree of freedom, with the Yates > correction >> > wi=st.GetValue("WinnersQty"); >> > Lo=st.GetValue("LosersQty"); >> > Chi = (abs(wi-Lo)-1)^2/(wi+Lo); >> > bo.AddCustomMetric( "Chi-Squared modif.: >10.83: very >> > significant(1000:1), >6.64: significant (100:1) , >3.84: probably >> > significant (20:1), <3.84: significance doubtful", Chi ); >> > >> > While this metric doesn't tell you anything if your system is >> > profitable, it tells you if its signals are only pure coincidence >> > (simply put). It's remarkable that many systems that seem to be >> > promising according to the usual metrics, are below 3.84, i.e. >> > significance doubtful. You need either a rather high number of >> trades >> > or a very high percentage of winning trades to shift this metric >> > significantly higher. At least for (medium-term) EOD systems >> (that's >> > what I trade) this is not easy to achieve. >> > >> > What do you think about this metric? Are there other "better" >> > statistical metrics? If yes - would you mind sharing the AFL code? >> > >> > Best regards, >> > >> > Thomas >> > >> > > > > > Please note that this group is for discussion between users only. > > To get support from AmiBroker please send an e-mail directly to > SUPPORT {at} amibroker.com > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > http://www.amibroker.com/devlog/ > > For other support material please check also: > http://www.amibroker.com/support.html > > Yahoo! Groups Links > > >
