Nick, using semideviation seems to be a self-evident approach. Any idea how to code that in AB?
Thanks Thomas On 08.10.2009, 18:01:36 NickW wrote: > Hi, > > This is a great thread. Thanks for all the valuable information. > > Using SQN is something I've struggled with aswell where is punishes the > outliers. When designing trend following systems, 5% of the trades are the > big winners that makes trend following work, so you really don't want to > punish the system for having few great winners. I am going to look more > into semideviation to see how that can solve that problem. > > Thanks > Nick > > On Thu, Oct 8, 2009 at 7:50 AM, Howard B <[email protected]> wrote: > > Hi Mike -- > > > > CAR is compound annual rate of return. Expectancy is the percentage (or > > dollar amount, but not for this discussion) gain on the average trade. > > The final value of the trading account, Terminal Relative Wealth in some > > descriptions, is: (1 + expectancy) raised to the power of the number of > > trades. When there are a few large trades included, this relationship is > > slightly different, but not enough to worry this discussion. CAR is: (1 > > + annual gain) raised to the power of the number of years. These are > > both based on geometric means. Aren't they the same, or so close that > > they act the same when used as an objective function? > > > > Van Tharp defines System Quality Number on page 28 of "Definitive Guide > > to Position Sizing." SQN = (expectancy / standard deviation) times > > squareroot of number of trades. SQN decreases when a median trade is > > replaced by either a large win or a large loss. It is possible to > > redefine the metric so that large wins are not penalized, such as by > > using the semi-deviation instead of the standard deviation as the > > denominator. But without making that change, outliers, both good and > > bad, do affect SQN by reducing it. > > > > What happens with open (or potentially open) trades at the boundaries of > > walk forward periods is difficult to handle in both theory and practice. > > Tomasz' implementation is to close all open trades at the end of each WF > > period; and go into each WF period flat, not taking a new position until > > there is a new signal. This creates a potentially serious distortion of > > results when trades are typically held a long time (a large proportion of > > a WF period) or when there are a few large trades that comprise the > > majority of a system's profit or loss. > > > > I think this leads to a conclusion that we both agree on -- high quality > > trading systems that can benefit from position sizing are based on high > > frequency trading with very careful control over losses, and even control > > over gains. > > > > Thanks for listening, > > Howard > > > > On Wed, Oct 7, 2009 at 2:36 PM, Mike <[email protected]> wrote: > >> Howard, > >> > >> Assuming that SQN is the t-test for expectancy, then optimizing on the > >> t-test of expectancy (i.e. SQN) is not the same as optimizing on CAR. > >> > >> The primary reason that CAR is a poor target for optimization is that > >> outliers can significantly improve the calculation. The exact opposite > >> is true for SQN. > >> > >> SQN rewards consistency and punishes outliers. Consistent winners with a > >> few large wins will improve CAR but hurt SQN, resulting in different > >> parameter combinations being selected during an optimization. > >> > >> As for writing a custom method, AmiBroker's stats are calculated based > >> on the assumption that all open trades are closed out at the backtest > >> boundary date. Many open trades, or even just a few large open trades, > >> can skew these values. > >> > >> For high frequency strategies or strategies using heavy position sizing, > >> creating a custom function is the only way to get reliable measurements. > >> > >> Mike > >> > >> > >> --- In [email protected] <amibroker%40yahoogroups.com>, Howard B > >> > >> <howardba...@...> wrote: > >> > Greetings all -- > >> > > >> > There has been a lot of activity on this thread. I'll not respond to > >> > >> each > >> > >> > point individually, but will make a couple of general comments. > >> > > >> > I know David Aronson, speak with him regularly, and collaborate with > >> > him > >> > >> on > >> > >> > projects. I have a copy of his book, "Evidence-Based Technical > >> > >> Analysis." > >> > >> > His book is excellent and I highly recommend it. I think David and I > >> > are > >> > >> in > >> > >> > pretty close agreement on most of the modeling, simulation, testing, > >> > and validation issues. > >> > > >> > I have spoken with Robert Pardo and have exchanged several emails and > >> > >> forum > >> > >> > postings with him. I think his earlier book was very good, > >> > particularly > >> > >> at > >> > >> > the time it was published. And his more recent book is not quite up to > >> > those standards. There are several important areas he did not cover > >> > and several areas where I see things considerably differently than > >> > Robert. > >> > > >> > I have spoken with and exchanged emails with Van Tharp, and I have > >> > >> copies of > >> > >> > his books "Trade Your Way to Financial Freedom" and "Definitive Guide > >> > to Position Sizing." Both are excellent, and I recommend them both > >> > highly. > >> > >> Be > >> > >> > sure to get the second edition of Trade Your Way to Financial Freedom > >> > -- > >> > >> it > >> > >> > has some important corrections and clarifications. > >> > > >> > Permit me a short rant on my soapbox. I really dislike it when people > >> > >> claim > >> > >> > ownership of common terms. Tom DeMark, Robert Pardo, Van Tharp, and > >> > >> others > >> > >> > put Service Mark symbols on terms that they think are unique to them, > >> > >> but > >> > >> > are not. I appreciate Tharp's enthusiasm over what he calls System > >> > >> Quality > >> > >> > Number, but I wish he would not put the Service Mark symbol next to > >> > >> every > >> > >> > occurrence of it. And trying to Service Mark the term Position Sizing > >> > is like a dietician service marking "calorie counting." Robert Pardo > >> > claims "Walk Forward." I used exactly that term describing exactly > >> > that process > >> > >> in > >> > >> > research papers I delivered at conferences in the late 1960s. The mark > >> > >> has > >> > >> > been registered, not by Robert, but by a company I used to work for > >> > and > >> > >> with > >> > >> > which Robert was not associated, over my strong objection. End of > >> > rant. > >> > > >> > System quality number is equivalent to t-test. Systems with SQNs above > >> > 2 work well for exactly the same reasons that systems with t-test > >> > scores > >> > >> above > >> > >> > 2 work well. In fact, it is possible to create a custom objective > >> > >> function > >> > >> > that Is the t-test and use it for optimization. Attendees at my > >> > >> workshops > >> > >> > in Melbourne later this month will see that demonstrated. Optimizing > >> > for the t-test of expectancy is equivalent to optimizing for CAR, so > >> > don't bother creating the custom function unless you have a better > >> > candidate > >> > >> for > >> > >> > your objective function than CAR. > >> > > >> > Back to the topic at hand ----- > >> > > >> > There is No rule of thumb to determine how long the in-sample period > >> > >> should > >> > >> > be. The Only way to determine that is by testing the model and the > >> > data together. And be prepared for that length to change over time. > >> > Some writers suggest a relationship between the number of free > >> > parameters and > >> > >> the > >> > >> > number of data points, or some proportional division of the available > >> > >> data. > >> > >> > Those techniques do work on industrial time-series data which is > >> > usually stationary, but they do not work on financial time-series data > >> > which is non-stationary and changes as trading systems become better > >> > at > >> > >> extracting > >> > >> > inefficiencies from it. > >> > > >> > No matter how good the in-sample results look, no matter how high the > >> > >> t-test > >> > >> > score is, no matter how many closed trades are represented -- > >> > in-sample results have no value in estimating the future performance > >> > of the > >> > >> system. > >> > >> > None. The only information you have that gives any indication of > >> > future performance are the out-of-sample results from testing on data > >> > that was never used at all -- not even once -- during system > >> > development. > >> > > >> > Tomorrow is out-of-sample. The only way to prepare for real-money > >> > >> trading > >> > >> > tomorrow is to be rigorous during the system testing and validation > >> > process. Anything less will overestimate the probability of success. > >> > > >> > Thanks for listening, > >> > Howard > ------------------------------------ **** IMPORTANT PLEASE READ **** This group is for the discussion between users only. This is *NOT* technical support channel. TO GET TECHNICAL SUPPORT send an e-mail directly to SUPPORT {at} amibroker.com TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at http://www.amibroker.com/feedback/ (submissions sent via other channels won't be considered) For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: http://www.amibroker.com/devlog/ Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/amibroker/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/amibroker/join (Yahoo! ID required) <*> To change settings via email: mailto:[email protected] mailto:[email protected] <*> To unsubscribe from this group, send an email to: [email protected] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
