Thanks you gentlemen, this conversation has been extremely informative. I commend you on keeping the scientific rigour and not let it obfuscate the communication.
best, Eswar 2009/7/15 brianw468 <[email protected]> > > > Hi Keith, > And many thanks for your excellent commonsense suggestions. Following this > up has proved to be a useful learning experience (including a fair amount of > re-learning!). > The first lesson is that one should always do some optimisation and > backtesting before leaping into a full-blown WF analysis. The second point > is that I should not expect max profit when optimising on a different > parameter - RAR/MDD in this case. However, I would not have expected the > profit to be too far down from the max - though this may be wishful > thinking!. > In case anyone is interested this is a summary of what I found by > optimising individual steps from the WF process:- > The detailed results and the 3-D graph for my IS1 period showed large > plateau areas, with over 100 cases giving the same (max) RAR/MDD. Most of > these had the same profit figure (533), but another group (still with the > same RAR/MDD) had a lower profit figure. Lesson no.3 was that my > optimisation steps were closer spaced than necessary. > Similarly, in IS2 a large number of cases had the same RAR/MDD (16.52) and > profit(474). Another group had RAR/MDD at 14.75 and P of 360. Then a third > set, and then a fourth with RAR/MDD at 13.14 and P at 1126. This was the > profit figure that showed up in the OOS1 results in the WF. It remains a > puzzle to me, because a Backtest in OOS1 using the optimum parameters from > IS1 gave a profit of 474. It might remain an unsolved mystery. > Finally, I guess that with a system like this that produces only a small > number of trades in a 6 month period, I should consider a more complex > fitness function - perhaps (e.g.) combining RAR/MDD with Profit in some way. > Thanks again > > Brian > > > --- In [email protected] <amibroker%40yahoogroups.com>, Keith > McCombs <kmcco...@...> wrote: > > > > Brain -- > > I don't know what the problem or the solution is. However, this should > > be fairly easy for you to debug. > > > > All you should have to do is repeat the walk forward process, one step > > at a time: > > 1. Optimize for IS1. Then using the resulting parameters, backtest > > IS1, and record the trades. > > 2. Using same parameters, backtest OOS1, and record the trades. > > 3. Compare with your WF results. You should have enough information > > now to find the problem. > > 4. If after 3 above, you still can't figure it out, then optimize OOS1 > > and IS2, which should produce identical results. If not, why not? > > > > The secret to debugging is "divide and conquer", just "one step at a > time". > > -- Keith > > > > brianw468 wrote: > > > > > > > > > Oops, > > > This message became a mess as Yahoo apparently removed most of the > > > spaces, destroying the layout.. > > > Results in the format (Sample, Profit, No. of Trades) are: > > > (IS1,533,1) (OOS1,1126,2) (IS2,474,1) (OOS2,-537,2) (IS3,43,1) > > > (OOS3,784,2). > > > I hope this comes out a bit clearer! > > > > > > Brian > > > > > > --- In [email protected] <amibroker%40yahoogroups.com><mailto: > amibroker%40yahoogroups.com <amibroker%2540yahoogroups.com>>, > > > "brianw468" <wild21@> wrote: > > > > > > > > Hi > > > > I recently tested a trading system but got some strange results from > > > the WF test. The WF process used exhaustive optimization. > > > > An extract from the results (too large to reproduce in full) is:- > > > > Item Profit No. Trades > > > > IS1 533 1 > > > > OOS1 1126 2 > > > > > > > > IS2 474 1 > > > > OOS2 -537 2 > > > > > > > > IS3 43 1 > > > > OOS 784 2 > > > > > > > > In two cases the OOS results are better than the preceding IS > > > results. Though a little surprising,this is quite plausible, because > > > the time periods differ. > > > > However,the IS2 profit is much poorer than the OOS1 profit, and this > > > is a worry since exhaustive optimization is used, and the time periods > > > are the same. Can anyone offer an explanation? Would it lie in the way > > > profit is handled when a trade is not completed by the end of the > > > selected period? > > > > > > > > Thanks > > > > Brian > > > > > > > > > > > > > > >
