Hi Brian again,
I have not read any of Ralph Vince but I will follow that lead up. This is a area I need some more theoretical understanding. Regards, Robert Melbourne, Au From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of brian_z111 Sent: Friday, 9 May 2008 11:37 To: [email protected] Subject: [amibroker] Re: AB Back-testing Metrics Hello Robert, No, this is an excellent question. The answer doens't come quickly though. I will do what I can in a short time. I specialise in system design/evaluation (SD&E) and MoneyManagement (MM) because IMO they are the most important part of freelance trading. I find that they have not been treated very will in the general trading literature (with some exceptions) - oversimplified, emphasis on the wrong points or just plain wrong. IMO this forum is doing a good job of SD&E with some people leading the charge and Herman and Fred specialists in optimizing and high freqency trading (which are closely related topics). I agree that MM is a subject that is underdone. So here is a starting point for your discussion on MM: - A trading strategy produces a trade series, in backtesting, which it the profile of that strategy. - stops etc help shape that profile - so the objective function/fitness measures (OF/FM) WE select will directly impact on the profile (we can, but don't have to include equity metrics in the OF/FM - that is a personal choice). - once we have selected a model when intend to trade then the SD&E phase is finished (for simplicity I will assume we are not going to continue with ongoing optimisation, for now). - my interest is in root cause evaluation and my argument is that the profile can most easily and usefully be described using some core metrics - the core metrics are binomial factors (because trading is analgous to a binomial event i.e. a coin flip - core metrics are W/L ratio, PayOff ratio, ave time in trade and ave time out of trade Note: these can be expressed in different ways - if we use $ values or % values, for example, we are assuming we are trading a fixed equity account or a compounding equity account - a metric like ProfitFactor is actually a standardised binomial average of all trades in your backtest (I don't actually favour PF entirely but it is a well known example). Now the important part: - the equity curve outcomes vary according to the MM rules we apply (as you noted) So, the profile is fixed (give or take variance) and is locked in at the SD&E phase The equity outcomes are dependent on the MM methods we use. Refer to RalphVince for the classic example of how underbetting will produce a smooth eq curve that doesn't go anywhere fast and how overbetting will send you broke (even if the coin is heavily biased in your favour) - I like his second book "The maths of MM etc" At the moment I am exporting the trade series and doing MM in excel. Later I will decide if I can do what I want to do within AB or not. If not,it is possible I will work on a MM plugin (years away?). There is too much going on for me to do it in AB at this stage but others, who have been down that path might be able to help there. My method: - export the trade series, of a validated system I intend to trade, as % Profit or Loss per trade - do MM in excel - the advantage is that without MM considerations within AB you can return all trades (regardless of whether you had the eq to take them or not) so you reach a statistically valid sample from a smaller dataset (which is a good thing). My proposition is that everything we can know, that is worth knowing, can be gleaned from the core metrics. This includes the trading edge we have (RV does a great job of showing us how to convert any trading edge into a winning business - of course the bigger the edge the better, relative to risk) I hope that helps somewhat. brian_z --- In [email protected] <mailto:amibroker%40yahoogroups.com> , "Robert Grigg" <[EMAIL PROTECTED]> wrote: > > I have been thinking through the process of evaluating the "goodness" of a > trading system using AB metrics and have become perplexed. Can someone who > has unravelled this issue previously help? > > There seem to be two general approaches to portfolio sizing while doing a > back-test. > > The first is to only back-test using the "Initial Equity" amount. > Generally, we might start using fixed position sizes and a fixed maximum > number of positions. In later developmental iterations we might use risk > based position sizing or other processes where we vary position sizing up to > the maximum amount of Initial Equity. I generally refer to this evaluation > approach as "Clamped Equity". This approach tends to give an equity curve > that is linear. > > The second approach is to compound profits and place trades up to "Current > Equity". (In AB terms our Position size is set to a % of Current Equity). > This is referred to as "Compounding Profits". The equity curve can take on > an exponential appearance. > > In real life trading most people tend to do a bit of both. However in > back-testing mode the "Compounding Profits" model (with a notionally good > system) can quickly become infeasible. (If only I had this system in > 2000...). > > So, now to the crux of the problem. The "Clamped Equity" approach, with a > notionally good system, produces a profit that is quarantined. Accumulated > profit can be used to top-up draw-downs but the amount in trades never > exceeds initial equity. In AmiBroker metrics, Exposure % is always > calculated on a bar by bar basis of mark-to-market holding against current > mark-to-market equity. However, in the "Clamped Equity" testing approach, > the quarantining of profits is intentional and it seems to me that it would > be more useful to look at the Exposure as a % of the "Clamped Equity" (i.e. > the "Initial Equity")? > > Exposure% is also used as a divisor in other metrics such as Net Risk > Adjusted Return %, Risk Adjusted Return %, Max System % Draw-down, > CAR/MaxDD and RAR/MaxDD and so these metrics also may be less useful given > this testing approach. > > I can see that comparisons between competing models, with the same test > period is valid. However, I do not feel so secure if I am doing > Walk-Forward back-testing using a complex objective function, particularly > if I am using weighted components that contain Exposure% and others that > don't. > > I know that it is relatively easy to use the Custom Back Tester to produce > amended statistics. However, I am concerned that I have not found any other > discussions of this issue on this or other forums, so maybe I have muddled > thinking and it is not a real issue. Any discussion would be appreciated. > > Robert >
