Howard, Assuming that SQN is the t-test for expectancy, then optimizing on the t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR.
The primary reason that CAR is a poor target for optimization is that outliers can significantly improve the calculation. The exact opposite is true for SQN. SQN rewards consistency and punishes outliers. Consistent winners with a few large wins will improve CAR but hurt SQN, resulting in different parameter combinations being selected during an optimization. As for writing a custom method, AmiBroker's stats are calculated based on the assumption that all open trades are closed out at the backtest boundary date. Many open trades, or even just a few large open trades, can skew these values. For high frequency strategies or strategies using heavy position sizing, creating a custom function is the only way to get reliable measurements. Mike --- In [email protected], Howard B <howardba...@...> wrote: > > Greetings all -- > > There has been a lot of activity on this thread. I'll not respond to each > point individually, but will make a couple of general comments. > > I know David Aronson, speak with him regularly, and collaborate with him on > projects. I have a copy of his book, "Evidence-Based Technical Analysis." > His book is excellent and I highly recommend it. I think David and I are in > pretty close agreement on most of the modeling, simulation, testing, and > validation issues. > > I have spoken with Robert Pardo and have exchanged several emails and forum > postings with him. I think his earlier book was very good, particularly at > the time it was published. And his more recent book is not quite up to > those standards. There are several important areas he did not cover and > several areas where I see things considerably differently than Robert. > > I have spoken with and exchanged emails with Van Tharp, and I have copies of > his books "Trade Your Way to Financial Freedom" and "Definitive Guide to > Position Sizing." Both are excellent, and I recommend them both highly. Be > sure to get the second edition of Trade Your Way to Financial Freedom -- it > has some important corrections and clarifications. > > Permit me a short rant on my soapbox. I really dislike it when people claim > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and others > put Service Mark symbols on terms that they think are unique to them, but > are not. I appreciate Tharp's enthusiasm over what he calls System Quality > Number, but I wish he would not put the Service Mark symbol next to every > occurrence of it. And trying to Service Mark the term Position Sizing is > like a dietician service marking "calorie counting." Robert Pardo claims > "Walk Forward." I used exactly that term describing exactly that process in > research papers I delivered at conferences in the late 1960s. The mark has > been registered, not by Robert, but by a company I used to work for and with > which Robert was not associated, over my strong objection. End of rant. > > System quality number is equivalent to t-test. Systems with SQNs above 2 > work well for exactly the same reasons that systems with t-test scores above > 2 work well. In fact, it is possible to create a custom objective function > that Is the t-test and use it for optimization. Attendees at my workshops > in Melbourne later this month will see that demonstrated. Optimizing for > the t-test of expectancy is equivalent to optimizing for CAR, so don't > bother creating the custom function unless you have a better candidate for > your objective function than CAR. > > Back to the topic at hand ----- > > There is No rule of thumb to determine how long the in-sample period should > be. The Only way to determine that is by testing the model and the data > together. And be prepared for that length to change over time. Some > writers suggest a relationship between the number of free parameters and the > number of data points, or some proportional division of the available data. > Those techniques do work on industrial time-series data which is usually > stationary, but they do not work on financial time-series data which is > non-stationary and changes as trading systems become better at extracting > inefficiencies from it. > > No matter how good the in-sample results look, no matter how high the t-test > score is, no matter how many closed trades are represented -- in-sample > results have no value in estimating the future performance of the system. > None. The only information you have that gives any indication of future > performance are the out-of-sample results from testing on data that was > never used at all -- not even once -- during system development. > > Tomorrow is out-of-sample. The only way to prepare for real-money trading > tomorrow is to be rigorous during the system testing and validation > process. Anything less will overestimate the probability of success. > > Thanks for listening, > Howard >
