Hi Keith, And many thanks for your excellent commonsense suggestions. Following this up has proved to be a useful learning experience (including a fair amount of re-learning!). The first lesson is that one should always do some optimisation and backtesting before leaping into a full-blown WF analysis. The second point is that I should not expect max profit when optimising on a different parameter - RAR/MDD in this case. However, I would not have expected the profit to be too far down from the max - though this may be wishful thinking!. In case anyone is interested this is a summary of what I found by optimising individual steps from the WF process:- The detailed results and the 3-D graph for my IS1 period showed large plateau areas, with over 100 cases giving the same (max) RAR/MDD. Most of these had the same profit figure (533), but another group (still with the same RAR/MDD) had a lower profit figure. Lesson no.3 was that my optimisation steps were closer spaced than necessary. Similarly, in IS2 a large number of cases had the same RAR/MDD (16.52) and profit(474). Another group had RAR/MDD at 14.75 and P of 360. Then a third set, and then a fourth with RAR/MDD at 13.14 and P at 1126. This was the profit figure that showed up in the OOS1 results in the WF. It remains a puzzle to me, because a Backtest in OOS1 using the optimum parameters from IS1 gave a profit of 474. It might remain an unsolved mystery. Finally, I guess that with a system like this that produces only a small number of trades in a 6 month period, I should consider a more complex fitness function - perhaps (e.g.) combining RAR/MDD with Profit in some way. Thanks again
Brian --- In [email protected], Keith McCombs <kmcco...@...> wrote: > > Brain -- > I don't know what the problem or the solution is. However, this should > be fairly easy for you to debug. > > All you should have to do is repeat the walk forward process, one step > at a time: > 1. Optimize for IS1. Then using the resulting parameters, backtest > IS1, and record the trades. > 2. Using same parameters, backtest OOS1, and record the trades. > 3. Compare with your WF results. You should have enough information > now to find the problem. > 4. If after 3 above, you still can't figure it out, then optimize OOS1 > and IS2, which should produce identical results. If not, why not? > > The secret to debugging is "divide and conquer", just "one step at a time". > -- Keith > > brianw468 wrote: > > > > > > Oops, > > This message became a mess as Yahoo apparently removed most of the > > spaces, destroying the layout.. > > Results in the format (Sample, Profit, No. of Trades) are: > > (IS1,533,1) (OOS1,1126,2) (IS2,474,1) (OOS2,-537,2) (IS3,43,1) > > (OOS3,784,2). > > I hope this comes out a bit clearer! > > > > Brian > > > > --- In [email protected] <mailto:amibroker%40yahoogroups.com>, > > "brianw468" <wild21@> wrote: > > > > > > Hi > > > I recently tested a trading system but got some strange results from > > the WF test. The WF process used exhaustive optimization. > > > An extract from the results (too large to reproduce in full) is:- > > > Item Profit No. Trades > > > IS1 533 1 > > > OOS1 1126 2 > > > > > > IS2 474 1 > > > OOS2 -537 2 > > > > > > IS3 43 1 > > > OOS 784 2 > > > > > > In two cases the OOS results are better than the preceding IS > > results. Though a little surprising,this is quite plausible, because > > the time periods differ. > > > However,the IS2 profit is much poorer than the OOS1 profit, and this > > is a worry since exhaustive optimization is used, and the time periods > > are the same. Can anyone offer an explanation? Would it lie in the way > > profit is handled when a trade is not completed by the end of the > > selected period? > > > > > > Thanks > > > Brian > > > > > > > >
