>people do a lot of silly things. A lot of them agree on these silly things True enough. This thing, though, has been the subject of so much academic research that it is probably not one of them. Of course, just like you, everybody wants to use as much data as possible for model optimization, so a technique called "bootstrapping" is used, which is similar to walk-forward optimization that I was proposing.
________________________________ From: ShaggsTheStud <[email protected]> To: [email protected] Sent: Tue, December 7, 2010 11:13:26 PM Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization I dunno, people do a lot of silly things. A lot of them agree on these silly things. If you had extra data, why would you not use it to see the sensitivity of your parameters? On Tue, Dec 7, 2010 at 8:34 PM, Astor <[email protected]> wrote: Shaggs, I wish I could claim credit for this approach but it is not my approach. It is a standard statistical methodology used by every professional Quant shop, without exceptions. In institutional settings, you could never get any strategy past the Investment Committee without presenting strong out-of-sample results. > >This is not to say that sensitivity to parameter changes, robustness checks, >etc need not be done. They still need to be done on in-sample data. > > > ________________________________ From: ShaggsTheStud <[email protected]> >To: [email protected] >Sent: Tue, December 7, 2010 8:53:49 PM > >Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization > > >So the difference between our approaches is that in your approach you look at >the 1 "optimal" case, and then you try it on some other set of data to verify >it. Ok, that is interesting. > >I much prefer to have one set of data, and look at the optimization map, and >view the sensitivities to changes in the parameters. A robust strategy will >not >be a picking out small "local minimums", it will have a wide plateau of >profitibility, and have good distribution on the trades graph where there are >not large periods of drawdown. > >I think my method is more robust, and would yield better real world >performance >than your method, but I can't prove it. > > >On Tue, Dec 7, 2010 at 5:52 PM, Astor <[email protected]> wrote: > > >>No, you do not do the same thing on both sets. You optimize and test >>different >>models on in-sample set only. You can do it as much as is necessary to get >>good >>results. You test only the final model on your out-of-sample and you can not >>change the model or re-optimize parameters and re-test on >>out-of-sample. Out-of-sample is like virginity, - once used it is gone. >> >>Results from out-of-sample is what you expect to get in real trading. >> >> ________________________________ From: ShaggsTheStud <[email protected]> >>To: [email protected] >>Sent: Tue, December 7, 2010 7:01:03 PM >> >>Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization >> >> >>Doing the same thing on two different sets of data seems identical to doing >>it >>on one combined set of data. How is it different? >> >> >>On Tue, Dec 7, 2010 at 4:14 AM, Astor <[email protected]> wrote: >> >>The "in-sample" set is where you develop your model and optimize your >>parameters. Because optimization searches through a very large number of >>possible parameter values, it finds those values which best fit the datain >>this >>set. In a different data set, such as the one that may occur in real trading, >>these parameters may prove perfectly useless. In Quant research, such >>situation >>is (derogatively) referred to as "datamining" or overfitting. With enough >>model >>parameters and extensive optimization, I can get perfect accuracy >>predicting "in-sample" lottery winners. Of course that model will not work to >>predict next, "out-of-sample", lottery winner. >>> >>>The "out-of-sample" set is a way to verify that the found model and its >>>parameters are general instead of unique to the "in-sample" development set. >>>Combining the two sets into a single set defeats that purpose. >>> >>> >>> ________________________________ From: ShaggsTheStud <[email protected]> >>>To: [email protected] >>>Sent: Mon, December 6, 2010 10:21:59 PM >>>Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization >>> >>> >>>That whole "in sample" and "out of sample" data thing strikes me very as >>>very >>>odd. If it works on the in-sample and not the out-sample, its going to have >>>a >>>bad distribution as a single set, so why not just combine it? >>> >>> >>>On Sun, Dec 5, 2010 at 5:56 AM, Astor <[email protected]> wrote: >>> >>>> we would >>>>>be required to significantly shorten our optimization periods, thus >>>>>incurring a penalty of standard error in our confidence bands. >>>> >>>>I understand your concern Eugene. However, it is important to recognize >>>>that >>>>in strategy development and validation there are two sets of data and two >>>>sets >>>>of confidence bands. First set is used for strategy development and >>>>parameter >>>>optimization and is often called "in-sample". The second set is used only >>>>to >>>>validate the strategy performance and is called "out-of-sample". >>>> >>>>If the confidence interval is very broad (standard error is large) in the >>>>"in-sample" data, your strategy is not reliable and should not be used. >>>> >>>>If the "in-sample" results are good and have acceptable confidence >>>>intervals, >>>>the next step is validation of the strategy on "out-of-sample" data. >>>>Because "out-of-sample" data has not been used for parameter optimization, >>>>the >>>>results obtained on this data are far more important than those from >>>>"in-sample". If the "out-of-sample confidence interval is too broad, the >>>>validation results are not reliable and the strategy should not be used. >>>> >>>>It is extremely common that the available data set is too small to >>>>partition the >>>>data into in- and out- of sample sets of adequate size. In financial >>>>research, the data set size is usually limited not by the data availability >>>>but >>>>by the data stationarity. To create valid sample sizes from small data, a >>>>technique called "leave-one-out" or "bootstrapping" or "jackknifing" is >>>>used. In >>>>those techniques the model is developed on the entire data except for one >>>>"holdout" point, then tested on this point. Then a different point is >>>>selected >>>>and the process is repeated. The validation results are obtained by >>>>combining >>>>the results of holdout points. Walk-forward optimization is an example of >>>>this >>>>technique and actually reduces standard error in the more >>>>important "out-of-sample" test. >>>> >>>> >>>> >>>>>better model would be the one which not only >>>>>accounts for the supply/demand, but also for its changing elasticity >>>>>over time >>>> >>>>That is definitely so and is often driven by seasonality as well as regime >>>>shifts. For futures, such as ES, the elasticity could drift in response to >>>>the proximity of the expiration date or as a result of changing market >>>>sentiment >>>>or increased trading in spot or in "dark pools", which impacts demand but >>>>is not >>>>reflected in bid/ask quotes. >>>> >>>> >>>>>the manner in which its parameters change overtime is not intuitive at >>>>>all >>>> >>>>If the value of the parameters themselves is not intuitive, then its change >>>>over >>>>time is very likely not to be intuitive as well and vice versa. Most >>>>non-intuitive parameter changes happen when the optimization surface is >>>>very >>>>flat or has many local maxima. Then a minor change in the data can put you >>>>into >>>>a very different local maxima and cause very unsettling parameter jumps. >>>>That is >>>>why restricting the optimization region to the vicinity of the most recent >>>>parameter values allows for parameters to only drift gradually. Then trends >>>>in >>>>parameter changes can be spotted and understood intuitively. >>>> >>>> >>>> ________________________________ From: nonlinear5 <[email protected]> >>>>To: JBookTrader <[email protected]> >>>>Sent: Sat, December 4, 2010 11:34:20 PM >>>>Subject: [JBookTrader] Re: Dynamic Parameter Optimization >>>> >>>> >>>>> Eugene, your comment goes to the need to have sufficiently large backtest >>>>> database relative to the number of adjustable parameters, so that the >>results >>>>> are statistically significant. How does that relate to potential >>>>> non-stationarity of parameters? >>>> >>>>The non-stationarity of parameters is a problem, indeed. However, some >>>>things are more or less absolute. Think of the supply/demand >>>>relationship. If you can capture its essence in the strategy, that >>>>should work today, tomorrow, and 10 years in the future. Now, I do >>>>acknowledge that a better model would be the one which not only >>>>accounts for the supply/demand, but also for its changing elasticity >>>>over time. However, such model would be more complex, more difficult >>>>to understand, and more time-consuming to test. Perhaps more >>>>importantly, while the supply/demand law by itself is quite intuitive, >>>>the manner in which its parameters change overtime is not intuitive at >>>>all. The best we can hope for in our walk-forward optimization is that >>>>whatever parameters were the "optimal" in a recent period would still >>>>be the optimal in the next period. For the sake of this hope, we would >>>>be required to significantly shorten our optimization periods, thus >>>>incurring a penalty of standard error in our confidence bands. >>>> >>>>-- >>>>You received this message because you are subscribed to the Google Groups >>>>"JBookTrader" group. >>>>To post to this group, send email to [email protected]. >>>>To unsubscribe from this group, send email to >>>>[email protected]. >>>>For more options, visit this group at >>>>http://groups.google.com/group/jbooktrader?hl=en. >>>> >>>> >>>> >>>>-- >>>>You received this message because you are subscribed to the Google Groups >>>>"JBookTrader" group. >>>>To post to this group, send email to [email protected]. >>>>To unsubscribe from this group, send email to >>>>[email protected]. >>>>For more options, visit this group at >>>>http://groups.google.com/group/jbooktrader?hl=en. >>>> >>>-- >>>You received this message because you are subscribed to the Google Groups >>>"JBookTrader" group. >>>To post to this group, send email to [email protected]. >>>To unsubscribe from this group, send email to >>>[email protected]. >>>For more options, visit this group at >>>http://groups.google.com/group/jbooktrader?hl=en. >>> >>> >>>-- >>>You received this message because you are subscribed to the Google Groups >>>"JBookTrader" group. >>>To post to this group, send email to [email protected]. >>>To unsubscribe from this group, send email to >>>[email protected]. >>>For more options, visit this group at >>>http://groups.google.com/group/jbooktrader?hl=en. >>> >>-- >>You received this message because you are subscribed to the Google Groups >>"JBookTrader" group. >>To post to this group, send email to [email protected]. >>To unsubscribe from this group, send email to >>[email protected]. >>For more options, visit this group at >>http://groups.google.com/group/jbooktrader?hl=en. >> >> >>-- >>You received this message because you are subscribed to the Google Groups >>"JBookTrader" group. >>To post to this group, send email to [email protected]. >>To unsubscribe from this group, send email to >>[email protected]. >>For more options, visit this group at >>http://groups.google.com/group/jbooktrader?hl=en. >> >-- >You received this message because you are subscribed to the Google Groups >"JBookTrader" group. >To post to this group, send email to [email protected]. >To unsubscribe from this group, send email to >[email protected]. >For more options, visit this group at >http://groups.google.com/group/jbooktrader?hl=en. > > >-- >You received this message because you are subscribed to the Google Groups >"JBookTrader" group. >To post to this group, send email to [email protected]. >To unsubscribe from this group, send email to >[email protected]. >For more options, visit this group at >http://groups.google.com/group/jbooktrader?hl=en. > -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en. -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.
