Thanks Mike, but I was wondering if anyone had explored weighting the in sample period of a walkforward test and if they found it to be useful.
My current custom fitness function invokes a dll that performs the bulk of the backtest on the gpu. Very fast, but a PITA to modify. --- In [email protected], "Mike" <[EMAIL PROTECTED]> wrote: > > AmiBroker supports the use of any backtest metric as the fitness > function, including custom metrics. As such, you can write your own > custom metric and use it as the fitness function. > > To add a custom metric: > http://www.amibroker.com/guide/a_custommetrics.html > > To make that metric the fitness function: > http://www.amibroker.com/kb/2008/02/12/getting-started-with- automatic- > walk-forward-optimization/ > > Notice the quote "You can also TYPE-IN any custom metric that you > have added via custom backtester interface". > > Mike > > --- In [email protected], "dloyer123" <dloyer123@> wrote: > > > > Very good point. > > > > Is there a good way to gradually decrease the impact of older data > on > > the system during walkforward data? > > > > The data expiring off the back end of the walkforward in sample > > period has just as much impact on the optimal values as the new > data > > collected today. > > > > Wouldn't it be better give more importance and less importance to > > older data. Would that not reduce the system "lag" in response to > > changing market conditions? > > > > For example, last aug was a very "interesting" time. With a 1 year > > in sample period, that data would be aging off now and could result > > in a step change to the optimal system values. > > > > If we could somehow use a fitness function that devalued older > traded > > compared to newer trades, it might make the system adapt more > quickly > > and avoid rapid changes due to old data aging out of the in sample > > period. > > > > > > > > --- In [email protected], "Howard B" <howardbandy@> wrote: > > > > > > Greetings all -- > > > > > > If the walk forward tests show poor out-of-sample results for the > > entire > > > out-of-sample period, that means the model is learning noise, not > > signal. > > > If they are good for a short period out-of-sample, then fall off, > > that means > > > the characteristics of the market being modeled are changing and > > you need > > > more frequent reoptimization -- a shorter out-of-sample time > period. > > > > > > If you have a model that has in the past been profitable out-of- > > sample, and > > > it becomes unprofitable during a period of real trading, you can > > reoptimize > > > ahead of schedule. In fact, if you have valid system, you can > > reoptimize as > > > frequently as you want to. You might want to run some walk > forward > > tests to > > > verify that this works for your system. If you have not tested > > early > > > reoptimizing, but do it to try to resync a system that is in a > > drawdown, I > > > recommend paper trading the new parameter set for a while to be > > certain that > > > the model just needed re-synchronizing, and was not terminally > > broken. > > > > > > Thanks, > > > Howard > > > > > > > > > > > > On Mon, Sep 8, 2008 at 2:05 PM, Mike <sfclimbers@> wrote: > > > > > > > Sometimes it is not so much adaptation that is needed. But > > rather just > > > > capturing the transition of cycles. > > > > > > > > Consider a house located in a region that has warm summers and > > cold > > > > winters. The variables would be climate control (e.g. air > > > > conditioner/heater). The variable values will change as the > > seasons > > > > unfold. To optimize comfort, we will be using increased air > > > > conditioning during the summers and increased heating during the > > > > winters. > > > > > > > > By experimenting with OOS time periods we may be able to > identify > > > > market cycles. If you only reoptimized your values once a year > > (e.g. > > > > set for average temperature for the year), your climate control > > would > > > > likely be generally too warm in the summers and generally too > > cold in > > > > the winters. But, if you reoptimized monthly, then you would > > gradually > > > > use increasing air conditioning leading into the summer and > > conversely > > > > gradually increase heating leading into the winter. > > > > > > > > Obviously the markets are not so perfectly cyclical as the > > seasons. > > > > But, the example illustrates the point. > > > > > > > > Mike > > > > > > > > > > > > --- In [email protected] <amibroker% > > 40yahoogroups.com>, "Ara > > > > Kaloustian" <ara1@> wrote: > > > > > > > > > > WF testing seems like a logical step to take in creating a > > trading > > > > system. While I am not suggesting that we don't do it, I'd like > to > > > > clear my understanding. > > > > > > > > > > It seems to me that we are looking to create a system with > > > > parameters that are valid "for all time". If a test is > succesful > > in > > > > OOS period, then we can assume that we have found something that > > > > remains applicable for the future - at least in this one > instance. > > > > > > > > > > Given that the markets change all the time, the obvious > > conclusion > > > > for me is that we need indicators that are adaptive... and that > > any > > > > that do not adapt will simply not work in OOS. > > > > > > > > > > The issue of providing feedback from equity curve seems valid > > and it > > > > can provide a warning about system starting to not perform as > > expected > > > > ... so regardless of type of indicators used, this kind of > > feedback is > > > > good!. > > > > > > > > > > The point I am making is that there has been very little said > > about > > > > adaptive indicators ... > > > > > > > > > > maybe that is the holy grail ... and therefore untennable ... > > > > > > > > > > > > > > > > > > > > > > >
