Completely agree. Have often wondered that myself. I am also not a statistician; instead I am (predominantly) a physicist who on occasion uses statistics. I always felt that working studying system performance in time is putting the horse behind the wagon. One is interested primarily in system activity (trades). Time only enters secondary to show how long the activity took or to determine the activity rate.
In fact it would make more sense to also deal with the IS in terms of number of trades. In essence you are replacing time units with (trade) activity units. In effect it is analogous to replacing time units with tick units. This approach has several advantages: You can 1. compare and optimize OS size (# of trades) with IS size (# of trades) 2. Determine what the optimum ratios are. 3. Easily see what the historical IS performance is per IS trade block (#of trades) 4. Easily see what the OS performance is per OS trade block (#of trades) 5. Determine the extent of deterioration of OS performance with # of trade blocks since the start of the OS Then indicators can be created displaying IS and/or OS below the charts and the chart background color can be alternated to display block changes. This way you have a little bit better idea under which circumstances the system works. From: [email protected] [mailto:[email protected]] On Behalf Of Herman vandenBergen Sent: Friday, February 19, 2010 7:51 AM To: Howard B Subject: Re: [amibroker] Walk Forward IS/OOS Period Optimization? Hello Howard, I cannot help but wonder why the OOS periods in optimizations aren't based on the number of trades, instead of some arbitrary time period. Could it be that this is just another example of how TA methodology is based on convenience, i.e., a reluctance to redesign tools and/or a reluctance to change :-) I am no statistician, but I would guess that if one wanted some specific OOS performance the first thing to do would be to figure out how many trades would be needed to do the calculations. If 100 trades gives you the confidence why would you run a OOS period that gives you 200 trades? Best regards, herman Friday, February 19, 2010, 8:18:45 AM, you wrote: Hi SpaceBass -- The only way to determine the correct length for the in-sample period is by running experiments. The length needs to be long enough for the model to synchronize with the data and learn to recognize the signal. But not so long that the signal has changed significantly, making it hard to identify. And not so short that there is not enough signal to learn, resulting in a system that has synced to the noise. In a few words -- the length of the in-sample period should be as short as is practical and effective. Once the length of the in-sample period has been determined, the length of the out-of-sample period is easy. It is the length of time that the system remains profitable. There is no general relationship between the anything and the length of the in-sample period. There is no relationship between the length of the in-sample period and the length of the out-of-sample period. There is some controversy in the modeling and simulation field about whether the in-sample length is a legitimate variable in an optimization. While it can be dangerous to system validity to run an optimization on in-sample length, some trials using different lengths are necessary. Thanks for listening, Howard On Thu, Feb 18, 2010 at 7:23 PM, spacebass5000 <[email protected]> wrote: I was wondering if there was a way to optimize the In-Sample and Step time periods within AB. If not, can someone point me in the direction of a good resource on this topic?
<<image001.gif>>
<<image002.gif>>
