Thanks for the contribute Howard, when I spoke about more or less 30 trades I meant trades inside the IS period, I fully agree that having few trades in OS period is not a problem
according to your huge experience do you believe that we have to consider a minimum number of trades in the IS period to have statistical value during the optimization of the parameters? for example: I have 5 trades in IS period --> try a longer IS period thanks --- In [email protected], Howard B <howardba...@...> wrote: > > Greetings all -- > > My point of view on the length of the in-sample and out-of-sample may be a > little different. > > The logic of the code has been designed to recognize some pattern or > characteristic of the data. The length of the in-sample period is however > long it takes to keep the model (the logic) in synchronization with the > data. There is no one answer to what that length is. When the pattern > changes, the model fits it less well. When the pattern changes > significantly, the model must be re-synchronized. The only person who can > say whether the length is correct or should be longer or shorter is the > person running the tests. > > The length of the out-of-sample period is however long the model and the > data remain in sync. That must be some length of time beyond the in-sample > period in order to make profitable trades. It could be a long time, in > which case there is no need to modify the model at all during that period. > There is no general relationship between the length of the in-sample period > and the length of the out-of-sample period -- none. There is no general > relationship between the performance in-sample and the performance > out-of-sample. The greater the difference between the two, the better the > system has been fit to the data over the in-sample period. But that does > not necessarily mean that the out-of-sample results are less meaningful. > > You can perform some experiments to see what the best in-sample length is. > And then to see what the typical out-of-sample length is. Knowing these > two, set up a walk forward run using those lengths. After the run is over, > ignore the in-sample results. They have no value in estimating the future > performance of the system. It is the out-of-sample results that can give > you some idea of how the system might act when traded with real money. > > It is nice to have a lot of closed traded in the out-of-sample period, but > you can run statistics on as few as 5 or 6. Having fewer trades means that > it will be more difficult to achieve statistical significance. The number > 30 is not magic -- it is just conventional. > > I think it helps to distinguish between the in-sample and out-of-sample > periods this way -- in-sample is seeing how well the model can be made to > fit the older data, out-of-sample is seeing how well it might fit future > data. > > Ignore the television ads where person after person exclaims "backtesting!" > as though that is the key to system development. It is not. Backtesting by > itself, without going on to walk forward testing, will give the trading > system developer the impression that the system is good. In-sample results > are always good. We do not stop fooling with the system until they are > good. But in-sample results have no value in predicting future performance > -- none. > > There are some general characteristics of trading systems that make them > easier to validate. Those begin with having a positive expectancy -- no > system can be profitable in the long term unless it has a positive > expectancy. Then going on to include trade frequently, hold a short time, > minimize losses. Of course, there have been profitable systems that trade > infrequently, hold a long time, and suffer deep drawdowns. It is much > harder to show that those were profitable because they were good rather than > lucky. > > There is more information about in-sample, out-of-sample, walk forward > testing, statistical validation, objective functions, and so forth in my > book, "Quantitative Trading Systems." > http://www.quantitativetradingsystems.com/ > > Thanks for listening, > Howard > > On Sun, Oct 4, 2009 at 10:56 AM, Bisto <bistoma...@...> wrote: > > > > > > > Yes, I believe that you should increase the IS period > > > > as general rule is not true "the shortest the best" trying to catch every > > market change because it's possible that a too short IS period produces a > > too low number of trades with no statistical robustness --> you will find > > parameters that are more likely candidated to fail in OS > > > > try a longer IS period and let's see what will happen > > > > I read an interesting book on this issue: "The evaluation and optimization > > of trading strategies" by Pardo. Maybe he repeated too much times the same > > concepts nevertheless I liked it > > > > if anyone could suggest a better book about this issue it would be very > > appreciated > > > > > > Bisto > > > > --- In [email protected] <amibroker%40yahoogroups.com>, "Gonzaga" > > <gonzagags@> wrote: > > > > > > Oh, sorry, I am lost in translation ... ;-) > > > Yes I meant trades of my IS period. > > > I've got about 70 trades in my IS period, three months. > > > BUT, I buy stocks in a multiposition way.This means, that my hole capital > > divides among several stocks purchased simultaneously. > > > So, in my statistics, I use to average my trades. When I use > > maxopenpositions=7, I use to average my results every 7 trades. > > > Considering that, my trades in three months are not 70, but less ( not > > exactly 70/7, but less than 70) > > > > > > If I use maxopenposition=1, which is, invest all my capital every trade, > > in three months I would have about 29 trades. > > > So I suppose I have to increase the IS period.. isn`t it? > > > > > > > > > --- In [email protected] <amibroker%40yahoogroups.com>, "Bisto" > > <bistoman73@> wrote: > > > > > > > > What do you mean with "I don't have many buyings and sellings"? > > > > > > > > If you have less than 30 trades in an IS period, IMHO, you are using a > > too short period due to not statistical robustness --> WFA is misleading, > > try a longer IS period > > > > > > > > Bisto > > > > > > > > --- In [email protected] <amibroker%40yahoogroups.com>, > > "Gonzaga" <gonzagags@> wrote: > > > > > > > > > > Thanks for the answers > > > > > To Keith McCombs : > > > > > > > > > > I use 3 months IS test and 1 month step, this is, 1 month OS test. My > > system is an end-of day-system, so I don't have many buyings and sellings.. > > > > > Perhaps I should make bigger the IS period? > > > > > > > > > > anyway, my parameter behaves well in any period. Of course it is an > > optimized variable, but it doesn't fail in ten years, in none of those ten > > years, over 500 stocks.. a very long period.. > > > > > So, couldn't it be better, on the long run, than the parameters > > optimized with the WF study? > > > > > (In fact, I am using it now, the optimized variable) > > > > > That's my real question.. > > > > > > > > > > To dloyer123: > > > > > I haven't understood the meaning of the Walk Forward Efficency, and > > seems interesting. > > > > > can you explain it better, please..? > > > > > > > > > > > > > > > > > > > > --- In [email protected] <amibroker%40yahoogroups.com>, > > "dloyer123" <dloyer123@> wrote: > > > > > > > > > > > > I have had similar experiences. I like to use WFT to estimate what > > Pardo call's his "Walk Forward Efficency", or the ratio of the out of sample > > WF profits to just optimizing over the entire time period. > > > > > > > > > > > > A good system should have as high a WFE as posible. Systems with a > > poor WFE tend to do poorly in live trading. > > > > > > > > > > > > If you have a parm set that works well over a long period of live > > trading, then you are doing well! > > > > > > > > > > > > > > > > > > > > > > > > >
