Hi, This is a great thread. Thanks for all the valuable information.
Using SQN is something I've struggled with aswell where is punishes the outliers. When designing trend following systems, 5% of the trades are the big winners that makes trend following work, so you really don't want to punish the system for having few great winners. I am going to look more into semideviation to see how that can solve that problem. Thanks Nick On Thu, Oct 8, 2009 at 7:50 AM, Howard B <[email protected]> wrote: > > > Hi Mike -- > > CAR is compound annual rate of return. Expectancy is the percentage (or > dollar amount, but not for this discussion) gain on the average trade. The > final value of the trading account, Terminal Relative Wealth in some > descriptions, is: (1 + expectancy) raised to the power of the number of > trades. When there are a few large trades included, this relationship is > slightly different, but not enough to worry this discussion. CAR is: (1 + > annual gain) raised to the power of the number of years. These are both > based on geometric means. Aren't they the same, or so close that they act > the same when used as an objective function? > > Van Tharp defines System Quality Number on page 28 of "Definitive Guide to > Position Sizing." SQN = (expectancy / standard deviation) times squareroot > of number of trades. SQN decreases when a median trade is replaced by > either a large win or a large loss. It is possible to redefine the metric > so that large wins are not penalized, such as by using the semi-deviation > instead of the standard deviation as the denominator. But without making > that change, outliers, both good and bad, do affect SQN by reducing it. > > What happens with open (or potentially open) trades at the boundaries of > walk forward periods is difficult to handle in both theory and practice. > Tomasz' implementation is to close all open trades at the end of each WF > period; and go into each WF period flat, not taking a new position until > there is a new signal. This creates a potentially serious distortion of > results when trades are typically held a long time (a large proportion of a > WF period) or when there are a few large trades that comprise the majority > of a system's profit or loss. > > I think this leads to a conclusion that we both agree on -- high quality > trading systems that can benefit from position sizing are based on high > frequency trading with very careful control over losses, and even control > over gains. > > Thanks for listening, > Howard > > > > On Wed, Oct 7, 2009 at 2:36 PM, Mike <[email protected]> wrote: > >> >> >> Howard, >> >> Assuming that SQN is the t-test for expectancy, then optimizing on the >> t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR. >> >> The primary reason that CAR is a poor target for optimization is that >> outliers can significantly improve the calculation. The exact opposite is >> true for SQN. >> >> SQN rewards consistency and punishes outliers. Consistent winners with a >> few large wins will improve CAR but hurt SQN, resulting in different >> parameter combinations being selected during an optimization. >> >> As for writing a custom method, AmiBroker's stats are calculated based on >> the assumption that all open trades are closed out at the backtest boundary >> date. Many open trades, or even just a few large open trades, can skew these >> values. >> >> For high frequency strategies or strategies using heavy position sizing, >> creating a custom function is the only way to get reliable measurements. >> >> Mike >> >> >> --- In [email protected] <amibroker%40yahoogroups.com>, Howard B >> <howardba...@...> wrote: >> > >> > Greetings all -- >> > >> > There has been a lot of activity on this thread. I'll not respond to >> each >> > point individually, but will make a couple of general comments. >> > >> > I know David Aronson, speak with him regularly, and collaborate with him >> on >> > projects. I have a copy of his book, "Evidence-Based Technical >> Analysis." >> > His book is excellent and I highly recommend it. I think David and I are >> in >> > pretty close agreement on most of the modeling, simulation, testing, and >> > validation issues. >> > >> > I have spoken with Robert Pardo and have exchanged several emails and >> forum >> > postings with him. I think his earlier book was very good, particularly >> at >> > the time it was published. And his more recent book is not quite up to >> > those standards. There are several important areas he did not cover and >> > several areas where I see things considerably differently than Robert. >> > >> > I have spoken with and exchanged emails with Van Tharp, and I have >> copies of >> > his books "Trade Your Way to Financial Freedom" and "Definitive Guide to >> > Position Sizing." Both are excellent, and I recommend them both highly. >> Be >> > sure to get the second edition of Trade Your Way to Financial Freedom -- >> it >> > has some important corrections and clarifications. >> > >> > Permit me a short rant on my soapbox. I really dislike it when people >> claim >> > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and >> others >> > put Service Mark symbols on terms that they think are unique to them, >> but >> > are not. I appreciate Tharp's enthusiasm over what he calls System >> Quality >> > Number, but I wish he would not put the Service Mark symbol next to >> every >> > occurrence of it. And trying to Service Mark the term Position Sizing is >> > like a dietician service marking "calorie counting." Robert Pardo claims >> > "Walk Forward." I used exactly that term describing exactly that process >> in >> > research papers I delivered at conferences in the late 1960s. The mark >> has >> > been registered, not by Robert, but by a company I used to work for and >> with >> > which Robert was not associated, over my strong objection. End of rant. >> > >> > System quality number is equivalent to t-test. Systems with SQNs above 2 >> > work well for exactly the same reasons that systems with t-test scores >> above >> > 2 work well. In fact, it is possible to create a custom objective >> function >> > that Is the t-test and use it for optimization. Attendees at my >> workshops >> > in Melbourne later this month will see that demonstrated. Optimizing for >> > the t-test of expectancy is equivalent to optimizing for CAR, so don't >> > bother creating the custom function unless you have a better candidate >> for >> > your objective function than CAR. >> > >> > Back to the topic at hand ----- >> > >> > There is No rule of thumb to determine how long the in-sample period >> should >> > be. The Only way to determine that is by testing the model and the data >> > together. And be prepared for that length to change over time. Some >> > writers suggest a relationship between the number of free parameters and >> the >> > number of data points, or some proportional division of the available >> data. >> > Those techniques do work on industrial time-series data which is usually >> > stationary, but they do not work on financial time-series data which is >> > non-stationary and changes as trading systems become better at >> extracting >> > inefficiencies from it. >> > >> > No matter how good the in-sample results look, no matter how high the >> t-test >> > score is, no matter how many closed trades are represented -- in-sample >> > results have no value in estimating the future performance of the >> system. >> > None. The only information you have that gives any indication of future >> > performance are the out-of-sample results from testing on data that was >> > never used at all -- not even once -- during system development. >> > >> > Tomorrow is out-of-sample. The only way to prepare for real-money >> trading >> > tomorrow is to be rigorous during the system testing and validation >> > process. Anything less will overestimate the probability of success. >> > >> > Thanks for listening, >> > Howard >> > >> >> > >
