Greetings all -- In-sample results and out-of-sample results can be, and usually are, very different in their characteristics.
My experience is that the ratio (OOS/(IS+OOS)), where these are the in-sample and out-of-sample results, is difficult to compute and often even difficult to define. I have not found it to be of value in estimating the future performance of the system. My understanding of Bob Pardo's book is that he feels that the length of the out-of-sample period can be determined by a calculation based on the length of the in-sample period. If that were true, then the ratio would be a little easier to compute. But, in my experience, there is no relationship between the length of the out-of-sample period and the length of the in-sample period. And gathering the data and performing the calculation of the ratio for some objective functions would be difficult. Isn't it the out-of-sample results we are trying to estimate? Do we care what the in-sample results look like? If the out-of-sample results are terrible, why bother computing the ratio. If the out-of-sample results are good, why bother computing the ratio -- how will that information be used to improve the system? And if it is used to modify the system, then the previously out-of-sample data has become in-sample. Do any of the forum member have examples they can contribute where computing the ratio is helpful? Thanks, Howard
