Hi Brian, and all -- You wrote: "Theoretically the OOS results should be better/worse than the IS tests on a 50/50 basis."
------------------------------------------ In my experience, the out-of-sample results are seldom better than the in-sample results. Here are a couple of reasons that might explain why. 1. In-sample, the system learns both the signal and a portion of the noise. In the out-of-sample period, even if the signal remains the same, the noise has changed, so the performance associated with the noise disappears. 2. The signal changes over time -- the model loses sync with the data. The signal in the out-of-sample period is different than the signal in the in-sample period, so the performance due to the signal declines. Thanks, Howard On Sun, May 10, 2009 at 7:37 PM, brian_z111 <[email protected]> wrote: > > > > Near Random ? > > You're kidding right ? ... I'd thought you had beeen around longer >than > that ... > > All of your points aren't entirely clear to me but: > > I'm talking about daily bars (say the US S&P500) - my testing shows that > non-randomness is not significant. > > Inefficiencies do exist but we have to look for them elsewhere. > Opportunities do exist but we have to exploit them in ways other than using > structural models. > > > > On another level the number of trades has no relevance in calculating > K-Ratio > > I take it you want me to think again about the K-ratio. > > However... K-ratio is a slope/stderror comparison ... so slope doesn't vary > with N or the order of the trades? .... stderror is stdev inclusive ... > stdev tails out in both directions with time? > > >and as calculated in AB is useless for anything but systems that >trade > constant dollar amounts ... > > I haven't delved into the K-ratio in detail. > I am not a fan of equity curve analysis so I haven't thought about the > K-ratio in depth. > > I noticed that Kestner based his calcs on a constant contract basis and > then used logN to equate compounded eqcurves to a constant contract basis > ... are you saying that there is a mistake in his thinking or math at that > point? > > I recall you mentioned this before but I didn't follow up because last time > I looked the K-ration in AB was exactly as written in KL's last book? > > How do you calculate it and how do you use it? > > >By definition that would imply no degree of success over a long >period of > time. > > With methods that are dependent on predicting the future? .... yes it does! > > > --- In [email protected] <amibroker%40yahoogroups.com>, ftone...@... > wrote: > > > > Near Random ? > > > > You're kidding right ? ... I'd thought you had beeen around longer then > that ... > > > > On another level the number of trades has no relevance in calculating > K-Ratio and as calculated in AB is useless for anything but systems that > trade constant dollar amounts ... By definition that would imply no degree > of success over a long period of time. > > > > ----- Original Message ----- > > From: brian_z11 > > Date: Sunday, May 10, 2009 8:58 pm > > Subject: [amibroker] Re: Expectancy - and related--specifically K-rato > > To: [email protected] <amibroker%40yahoogroups.com> > > > > > My understanding of Bob Pardo's book is that he feels > > > that the length of the > > > out-of-sample period can be determined by a calculation based on > > > the length > > > of the in-sample period. If that were true, then the ratio > > > would be a > > > little easier to compute. But, in my experience, there is no > > > relationshipbetween the length of the out-of-sample period and > > > the length of the > > > in-sample period. And gathering the data and performing the > > > calculation of > > > the ratio for some objective functions would be difficult. > > > > > > I don't believe that it would be worth the effort but setting > > > tbe BT the task of collecting the same number of trades in the > > > OOS test as were collected in the IS test is the way to > > > standardise and simplify calculating this ratio. > > > > > > N == the number of closed trades in a test run; > > > Where IS(N) == OOS(N) the ratio IS(N bars)/OOS(N bars) will > > > always be defined as a probability distribution. > > > > > > No constant relationsip, between IS and OOS length (in bars) can > > > exist because of the (near) random nature of the markets. > > > > > > Believe that such a relationship does exist, and hence can be > > > calculated/exploited, is an artefact of the belief that the > > > markets are non random i.e. have a structure that we can > > > identify, model and therefore use to make predictions about the > > > future of the markets. > > > > > > Any attempt to improve OOS performance by 'tuning' the length of > > > the sample periods is a futile excercise. > > > > > > The exception to this, as you point out in your book, is that > > > the markets do change (they are not quite random .... human > > > behaviour leaves a faint trail of non randomness) and so current > > > data may be more relevant than non- current data e.g. increased > > > use of computers, and access to information, by traders may have > > > changed the markets in the last decade or two. > > > > > > Whether these changes are strong enough, and occur over short > > > enough time periods or identifiable time periods, to justify the > > > view that we can/must account for this, in our design processes, > > > is arguable i.e. it is questionable that we can "synchronise the > > > market and our trading system by shortening (varying) the time > > > periods used in our walk forward testing", as you suggested in > > > your QTS book (P261). > > > > > > Granted that we are attempting to synchronise our systems to > > > market behaviour but this is done within our system rules, not > > > by altering the N bars tested. > > > > > > The fundamental premise of trading is that the system will > > > perform, in the future, irrespective of market conditions or the > > > time period involved. > > > > > > The OOS walk through is a test to find out if the patterns we > > > identified in the IS data still exist in the future and that the > > > code we used to synchronise to that pattern(s) is effective. > > > > > > The only time that will be time dependent is when the patterns > > > are time based e.g. the number of times the price of oil goes up > > > on the first day of the month is statistically significant. > > > > > > Very few exploitable inefficiencies in the markets are time based. > > > > > > > > > On another point: > > > > > > Theoretically the OOS results should be better/worse than the IS > > > tests on a 50/50 basis. > > > > > > Continual skewing of that ratio to the downside is an indicator > > > of curve fitting at the IS stage? > > > > > > > > > > > > > > > > > > --- In [email protected] <amibroker%40yahoogroups.com>, Howard > B wrote: > > > > > > > > Greetings all -- > > > > > > > > In-sample results and out-of-sample results can be, and > > > usually are, very > > > > different in their characteristics. > > > > > > > > My experience is that the ratio (OOS/(IS+OOS)), where these > > > are the > > > > in-sample and out-of-sample results, is difficult to compute > > > and often even > > > > difficult to define. I have not found it to be of value in > > > estimating the > > > > future performance of the system. > > > > > > > > My understanding of Bob Pardo's book is that he feels that the > > > length of the > > > > out-of-sample period can be determined by a calculation based > > > on the length > > > > of the in-sample period. If that were true, then the ratio > > > would be a > > > > little easier to compute. But, in my experience, there is no > > > relationship> between the length of the out-of-sample period and > > > the length of the > > > > in-sample period. And gathering the data and performing the > > > calculation of > > > > the ratio for some objective functions would be difficult. > > > > > > > > Isn't it the out-of-sample results we are trying to estimate? > > > Do we care > > > > what the in-sample results look like? If the out-of-sample > > > results are > > > > terrible, why bother computing the ratio. If the out-of- > > > sample results are > > > > good, why bother computing the ratio -- how will that > > > information be used to > > > > improve the system? And if it is used to modify the system, > > > then the > > > > previously out-of-sample data has become in-sample. > > > > > > > > Do any of the forum member have examples they can contribute > > > where computing > > > > the ratio is helpful? > > > > > > > > Thanks, > > > > Howard > > > > > > > > > > > > > > > > > >
