Hi Howard, I follow this discussion with interest and I thank you for your invaluable comments. I am still confused about the statement .."look for systems that have truly out-of-sample t-test scores of 2.0".
In AB terms, what is the objective function we need to compare it to "t-test"? Are we looking for CAR/ADD to be greater than 2.0? Kind Regards Richard --- In [email protected], Howard B <howardba...@...> wrote: > > Hi Mike, and all -- > > There is a lot to like in Tharp's book, Definitive Guide to Position > Sizing. > > But it appears to me that he developed the concept of system quality number > without using it as an objective function used in testing and validating > trading systems. He describes several idealized trading systems that have a > range of SQNs -- from negative to about 7. One on page 38 is described as: > Mean expectancy: 3.42 > Standard deviation: 4.89 > Win percentage: 90 > Win/Loss Ratio: 2.35 > Number Trades: 100 > SQN: 6.99 > > Plugging those numbers into the formula for the t-test, which is > (mean/stdev) * sqrt(N) > we get (3.42/4.89)*10 which equals 6.99 -- the same as the SQN. > > While he does not call SQN t-test, for each of the examples I worked out, > the two are the same. > > So, one of my points is that SQN is equivalent to t-test where the > hypothesis that is being tested is "is expectancy greater than 0?" > > I like the idea that the statistical test is incorporated into the metric. > I have been using that concept for some time, and I think it works well. I > wish he had not called it SQN, and I wish he had not put a Service Mark > next to it every time it appears in his book. But that concept is good. > > I seriously doubt that he, or anyone else, has a series of truly > out-of-sample closed trades where the t-test of expectancy is 7. It would > take only a few weeks to turn $10,000 into enough to buy Manhattan with a > system like that. Which goes to my point that I doubt that any of the > idealized trading system he describes with large SQNs reflect real trading > systems. Idealized, yes. In-sample, perhaps, but I doubt it. > Out-of-sample, almost impossible and I very seriously doubt it. > > As a basis for my comment, look at a table of t-test percentile values. I > use a rule of thumb that the t-test should be above about 2.0, just for its > ease of computation. With n=10 trades (9 degrees of freedom), 95% > confidence comes with t-statistic of 1.83. With n=120, 99.5% confidence > comes with a t-statistic of 2.62. > > For sample sizes greater than about 60, the t-distribution and the normal > distribution are essentially identical. Something 3 standard deviations > away from the mean happens much much less than 1% of the time. Something 7 > standard deviations away is so rare as to be essentially impossible on our > time scale. It is nice to see the fantasy equity curve, but it will never > happen. > > If expectancy works as an objective function for a particular trader and > system, then by all means, use it. As the characteristics of the trade > returns change, the advantages and disadvantages of using expectancy change. > > All of that said, I think Tharp's book is good. It brings out many > excellent points about the importance of positions sizing. Read it in > conjunction with Ralph Vince's "The Leverage Space Trading Model." My own > research and experience is in agreement with both of those. > > There is an implication -- one that many people will not find comfortable > and will argue strongly against. It is this: The only way to generate > large increases in equity safely is to have a trading system that trades > frequently, holds a short period of time, and carefully limits losing > trades. > > Yes, there are examples of people who have made a lot of money trading > infrequently and holding long periods. My contention is that > 1. we are hearing from those who were successful. Not many people who > tried to do this succeeded. Most who tried and failed have not remained in > the trading community and we are not hearing their stories of failure. > 2. The period from 1982 to 2000, or even 2007, in US equities, is probably > a once in a millenium event. I doubt that it will be repeated -- I am > certain it will not be repeated in my lifetime. Long-only, buy-and-hold > systems did very well during that period. At the risk of being flip, it was > easy to confuse brains with a bull market. > > The price earning ratio for the S&P 500 companies is now about 120 (per > S&P's numbers on their website). The price earnings ratio at equity market > bottoms is single digits, and in normal times is mid-teens. Excursions to > an extreme tend to correct, and the correction seldom stops at the mean -- > it makes an excursion to the other extreme. Can we expect fundamentals to > improve enough so that earnings increase by a factor of 10? Or will prices > decrease by a factor of 10? Or some of each? I expect that one or more of > those events will be taking place in the next twelve months. In any event, > 10 percent per year annual increases in stock prices seem extremely > unlikely. We need trading systems that do not depend on a rising market. > > I suggest that we read Tharp and Vince, use good modeling and simulation > technique, test the significance of our out-of-sample results, and look for > systems that have truly out-of-sample t-test scores of 2.0 or more -- they > are hard enough to find -- forget 6.99. Then work on position sizing > methods that use anti-martingale techniques (increase position size when > winning, reduce it when losing) to determine what position size will produce > the goal the trader wants while keeping the probability of bankruptcy at a > low enough level so he or she is confortable trading. > > There are two very distinct components to this: > 1. Develop good trading systems that have positive expectancy, limiting the > losing trades to small amounts, and trade frequently. > 2. Apply aggressive position sizing when winning. > > > Thanks for listening, > Howard > > > > > > > > On Thu, Oct 8, 2009 at 11:15 PM, Mike <sfclimb...@...> wrote: > > > > > > > Hi Howard, > > > > I'm a bit confused by your remarks. > > > > 1. You have said that Definitive Guide To Positoin Sizing is an excellent > > book. > > 2. You have said that SQN is a t-test (of expectancy?). > > 3. You have said that a t-test of expectancy is functionally equivalent to > > CAR. > > 4. You have said that CAR is a poor fitness function. > > > > Given that the book repeatedly promotes usage of SQN as a comparator (i.e. > > a fitness function) with which to measure two systems, how do you reconcile > > point 1 with points 2-4? > > > > Just to pin it down without any indirection, are you saying that you > > consider SQN as poor a fitness function as CAR? Or, have I misunderstood > > something you've said? > > > > Thanks, > > > > > > Mike > > > > --- In [email protected] <amibroker%40yahoogroups.com>, Howard B > > <howardbandy@> wrote: > > > > > > Hi Mike -- > > > > > > CAR is compound annual rate of return. Expectancy is the percentage (or > > > dollar amount, but not for this discussion) gain on the average trade. > > The > > > final value of the trading account, Terminal Relative Wealth in some > > > descriptions, is: (1 + expectancy) raised to the power of the number of > > > trades. When there are a few large trades included, this relationship is > > > slightly different, but not enough to worry this discussion. CAR is: (1 + > > > annual gain) raised to the power of the number of years. These are both > > > based on geometric means. Aren't they the same, or so close that they act > > > the same when used as an objective function? > > > > > > Van Tharp defines System Quality Number on page 28 of "Definitive Guide > > to > > > Position Sizing." SQN = (expectancy / standard deviation) times > > squareroot > > > of number of trades. SQN decreases when a median trade is replaced by > > > either a large win or a large loss. It is possible to redefine the metric > > > so that large wins are not penalized, such as by using the semi-deviation > > > instead of the standard deviation as the denominator. But without making > > > that change, outliers, both good and bad, do affect SQN by reducing it. > > > > > > What happens with open (or potentially open) trades at the boundaries of > > > walk forward periods is difficult to handle in both theory and practice. > > > Tomasz' implementation is to close all open trades at the end of each WF > > > period; and go into each WF period flat, not taking a new position until > > > there is a new signal. This creates a potentially serious distortion of > > > results when trades are typically held a long time (a large proportion of > > a > > > WF period) or when there are a few large trades that comprise the > > majority > > > of a system's profit or loss. > > > > > > I think this leads to a conclusion that we both agree on -- high quality > > > trading systems that can benefit from position sizing are based on high > > > frequency trading with very careful control over losses, and even control > > > over gains. > > > > > > Thanks for listening, > > > Howard > > > > > > > > > On Wed, Oct 7, 2009 at 2:36 PM, Mike <sfclimbers@> wrote: > > > > > > > > > > > > > > > Howard, > > > > > > > > Assuming that SQN is the t-test for expectancy, then optimizing on the > > > > t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR. > > > > > > > > The primary reason that CAR is a poor target for optimization is that > > > > outliers can significantly improve the calculation. The exact opposite > > is > > > > true for SQN. > > > > > > > > SQN rewards consistency and punishes outliers. Consistent winners with > > a > > > > few large wins will improve CAR but hurt SQN, resulting in different > > > > parameter combinations being selected during an optimization. > > > > > > > > As for writing a custom method, AmiBroker's stats are calculated based > > on > > > > the assumption that all open trades are closed out at the backtest > > boundary > > > > date. Many open trades, or even just a few large open trades, can skew > > these > > > > values. > > > > > > > > For high frequency strategies or strategies using heavy position > > sizing, > > > > creating a custom function is the only way to get reliable > > measurements. > > > > > > > > Mike > > > > > > > > > > > > --- In [email protected] > > > > <amibroker%40yahoogroups.com><amibroker% > > 40yahoogroups.com>, Howard B > > > > > > <howardbandy@> wrote: > > > > > > > > > > Greetings all -- > > > > > > > > > > There has been a lot of activity on this thread. I'll not respond to > > each > > > > > point individually, but will make a couple of general comments. > > > > > > > > > > I know David Aronson, speak with him regularly, and collaborate with > > him > > > > on > > > > > projects. I have a copy of his book, "Evidence-Based Technical > > Analysis." > > > > > His book is excellent and I highly recommend it. I think David and I > > are > > > > in > > > > > pretty close agreement on most of the modeling, simulation, testing, > > and > > > > > validation issues. > > > > > > > > > > I have spoken with Robert Pardo and have exchanged several emails and > > > > forum > > > > > postings with him. I think his earlier book was very good, > > particularly > > > > at > > > > > the time it was published. And his more recent book is not quite up > > to > > > > > those standards. There are several important areas he did not cover > > and > > > > > several areas where I see things considerably differently than > > Robert. > > > > > > > > > > I have spoken with and exchanged emails with Van Tharp, and I have > > copies > > > > of > > > > > his books "Trade Your Way to Financial Freedom" and "Definitive Guide > > to > > > > > Position Sizing." Both are excellent, and I recommend them both > > highly. > > > > Be > > > > > sure to get the second edition of Trade Your Way to Financial Freedom > > -- > > > > it > > > > > has some important corrections and clarifications. > > > > > > > > > > Permit me a short rant on my soapbox. I really dislike it when people > > > > claim > > > > > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and > > > > others > > > > > put Service Mark symbols on terms that they think are unique to them, > > but > > > > > are not. I appreciate Tharp's enthusiasm over what he calls System > > > > Quality > > > > > Number, but I wish he would not put the Service Mark symbol next to > > every > > > > > occurrence of it. And trying to Service Mark the term Position Sizing > > is > > > > > like a dietician service marking "calorie counting." Robert Pardo > > claims > > > > > "Walk Forward." I used exactly that term describing exactly that > > process > > > > in > > > > > research papers I delivered at conferences in the late 1960s. The > > mark > > > > has > > > > > been registered, not by Robert, but by a company I used to work for > > and > > > > with > > > > > which Robert was not associated, over my strong objection. End of > > rant. > > > > > > > > > > System quality number is equivalent to t-test. Systems with SQNs > > above 2 > > > > > work well for exactly the same reasons that systems with t-test > > scores > > > > above > > > > > 2 work well. In fact, it is possible to create a custom objective > > > > function > > > > > that Is the t-test and use it for optimization. Attendees at my > > workshops > > > > > in Melbourne later this month will see that demonstrated. Optimizing > > for > > > > > the t-test of expectancy is equivalent to optimizing for CAR, so > > don't > > > > > bother creating the custom function unless you have a better > > candidate > > > > for > > > > > your objective function than CAR. > > > > > > > > > > Back to the topic at hand ----- > > > > > > > > > > There is No rule of thumb to determine how long the in-sample period > > > > should > > > > > be. The Only way to determine that is by testing the model and the > > data > > > > > together. And be prepared for that length to change over time. Some > > > > > writers suggest a relationship between the number of free parameters > > and > > > > the > > > > > number of data points, or some proportional division of the available > > > > data. > > > > > Those techniques do work on industrial time-series data which is > > usually > > > > > stationary, but they do not work on financial time-series data which > > is > > > > > non-stationary and changes as trading systems become better at > > extracting > > > > > inefficiencies from it. > > > > > > > > > > No matter how good the in-sample results look, no matter how high the > > > > t-test > > > > > score is, no matter how many closed trades are represented -- > > in-sample > > > > > results have no value in estimating the future performance of the > > system. > > > > > None. The only information you have that gives any indication of > > future > > > > > performance are the out-of-sample results from testing on data that > > was > > > > > never used at all -- not even once -- during system development. > > > > > > > > > > Tomorrow is out-of-sample. The only way to prepare for real-money > > trading > > > > > tomorrow is to be rigorous during the system testing and validation > > > > > process. Anything less will overestimate the probability of success. > > > > > > > > > > Thanks for listening, > > > > > Howard > > > > > > > > > > > > > > > > > > > > > > > > > > >
