it is online in the group file repository. Klaus
On 10 Dez., 14:37, Astor <[email protected]> wrote: > Thanks Klaus! I look forward to seeing it. > > ________________________________ > From: Klaus <[email protected]> > To: JBookTrader <[email protected]> > Sent: Fri, December 10, 2010 3:55:37 AM > Subject: [JBookTrader] Re: Dynamic Parameter Optimization > > Dear Astor, > > I posted it earlier for a previous JBT version. > I will upload a version for the current JBT in a few moments, This is > an excerpt of my production code, but note that I modified it somewhat > to > ignore certain error conditions (missing volume values and wrongly > ordered dates). > > Cheers > Klaus > > On 8 Dez., 21:53, Astor <[email protected]> wrote: > > > > > > > > > Thanks Klaus. Exellent description. Can you post your JBT extension for > > backtests? I would love to incorporate that into my program. > > > ________________________________ > > From: Klaus <[email protected]> > > To: JBookTrader <[email protected]> > > Sent: Wed, December 8, 2010 2:48:13 PM > > Subject: [JBookTrader] Re: Dynamic Parameter Optimization > > > use of-out-of sample data is a must in all machine learning approaches > > (and this is actually what we do here). > > (So, yes i also take the approach, this is actually the reason why I > > built JBT extension for batches of backtests.) > > > Perhaps it can be better understood if looks at the danger of > > presenting all the data. > > What can happen is that the strategy (and actually JBT does not > > support learning of parameters, but only of parameter settings) > > that is generated is sort of memorizing the presented data.. and then > > provides good results there, but not beyond. > > The result does not generalize to further data. > > > It is like with people, if you really want someone to understand > > something, you will teach him (presenting examples > > is one approach for teaching). But at the end you want to know whether > > he really understood (i.e., got the principles > > and is able to use them to solve knew problems) or whether he just > > memorized. The only way to find this out is to show him s.th. he has > > not seen before... > > That is what use of out-of-sample means. Simulated forward-Trading is > > another way to achieve the same result, but then you are doing it in > > real time (i.e., need weeks), but if you have more data, you can do > > this simply in minutes with testing.. > > > On 8 Dez., 15:15, Astor <[email protected]> wrote: > > > > >people do a lot of silly things. A lot of them agree on these silly > things > > > > > > True enough. This thing, though, has been the subject of so much academic > > > research that it is probably not one of them. Of course, just like > > > you, everybody wants to use as much data as possible for model > > > optimization, > > so > > > a technique called "bootstrapping" is used, which is similar to > > > walk-forward > > > optimization that I was proposing. > > > > ________________________________ > > > From: ShaggsTheStud <[email protected]> > > > To: [email protected] > > > Sent: Tue, December 7, 2010 11:13:26 PM > > > Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization > > > > I dunno, people do a lot of silly things. A lot of them agree on these > silly > > > things. > > > > If you had extra data, why would you not use it to see the sensitivity of > >your > > > parameters? > > > > On Tue, Dec 7, 2010 at 8:34 PM, Astor <[email protected]> wrote: > > > > Shaggs, I wish I could claim credit for this approach but it is not my > > >approach. > > > It is a standard statistical methodology used by every professional Quant > > shop, > > > without exceptions. In institutional settings, you could never get any > > strategy > > > past the Investment Committee without presenting strong out-of-sample > >results. > > > > >This is not to say that sensitivity to parameter changes, robustness > checks, > > > >etc need not be done. They still need to be done on in-sample data. > > > > ________________________________ > > > From: ShaggsTheStud <[email protected]> > > > > >To: [email protected] > > > >Sent: Tue, December 7, 2010 8:53:49 PM > > > > >Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization > > > > >So the difference between our approaches is that in your approach you > > > >look > >at > > > >the 1 "optimal" case, and then you try it on some other set of data to > >verify > > > >it. Ok, that is interesting. > > > > >I much prefer to have one set of data, and look at the optimization map, > and > > > >view the sensitivities to changes in the parameters. A robust strategy > will > > >not > > > >be a picking out small "local minimums", it will have a wide plateau of > > > >profitibility, and have good distribution on the trades graph where there > >are > > > >not large periods of drawdown. > > > > >I think my method is more robust, and would yield better real world > > >performance > > > >than your method, but I can't prove it. > > > > >On Tue, Dec 7, 2010 at 5:52 PM, Astor <[email protected]> wrote: > > > > >>No, you do not do the same thing on both sets. You optimize and test > > >different > > > >>models on in-sample set only. You can do it as much as is necessary to > > > >>get > > >good > > > >>results. You test only the final model on your out-of-sample and you can > >not > > > >>change the model or re-optimize parameters and re-test on > > > >>out-of-sample. Out-of-sample is like virginity, - once used it is gone. > > > > >>Results from out-of-sample is what you expect to get in real trading. > > > > ________________________________ > > > From: ShaggsTheStud <[email protected]> > > > > >>To: [email protected] > > > >>Sent: Tue, December 7, 2010 7:01:03 PM > > > > >>Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization > > > > >>Doing the same thing on two different sets of data seems identical to > doing > > >it > > > >>on one combined set of data. How is it different? > > > > >>On Tue, Dec 7, 2010 at 4:14 AM, Astor <[email protected]> wrote: > > > > >>The "in-sample" set is where you develop your model and optimize your > > > >>parameters. Because optimization searches through a very large number of > > > >>possible parameter values, it finds those values which best fit the > > > >>datain > > >this > > > >>set. In a different data set, such as the one that may occur in real > > trading, > > > >>these parameters may prove perfectly useless. In Quant research, such > > >situation > > > >>is (derogatively) referred to as "datamining" or overfitting. With > > > >>enough > > >model > > > >>parameters and extensive optimization, I can get perfect accuracy > > > >>predicting "in-sample" lottery winners. Of course that model will not > > > >>work > > to > > > >>predict next, "out-of-sample", lottery winner. > > > > >>>The "out-of-sample" set is a way to verify that the found model and its > > > >>>parameters are general instead of unique to the "in-sample" development > > set. > > > >>>Combining the two sets into a single set defeats that purpose. > > > > ________________________________ > > > From: ShaggsTheStud <[email protected]> > > > > >>>To: [email protected] > > > >>>Sent: Mon, December 6, 2010 10:21:59 PM > > > >>>Subject: Re: [JBookTrader] Re: Dynamic Parameter Optimization > > > > >>>That whole "in sample" and "out of sample" data thing strikes me very > > > >>>as > > >very > > > >>>odd. If it works on the in-sample and not the out-sample, its going to > >have > > >a > > > >>>bad distribution as a single set, so why not just combine it? > > > > >>>On Sun, Dec 5, 2010 at 5:56 AM, Astor <[email protected]> wrote: > > > > >>>> we would > > > >>>>>be required to significantly shorten our optimization periods, thus > > > >>>>>incurring a penalty of standard error in our confidence bands. > > > > >>>>I understand your concern Eugene. However, it is important to > > > >>>>recognize > > >that > > > >>>>in strategy development and validation there are two sets of data and > two > > >sets > > > >>>>of confidence bands. First set is used for strategy development and > > >parameter > > > >>>>optimization and is often called "in-sample". The second set is used > only > > >to > > > >>>>validate the strategy performance and is called "out-of-sample". > > > > >>>>If the confidence interval is very broad (standard error is large) in > the > > > >>>>"in-sample" data, your strategy is not reliable and should not be > > > >>>>used. > > > > >>>>If the "in-sample" results are good and have acceptable confidence > > >intervals, > > > >>>>the next step is validation of the strategy on "out-of-sample" data. > > > >>>>Because "out-of-sample" data has not been used for parameter > >optimization, > > >the > > > >>>>results obtained on this data are far more important than those from > > > >>>>"in-sample". If the "out-of-sample confidence interval is too broad, > > > >>>>the > > > >>>>validation results are not reliable and the strategy should not be > > > >>>>used. > > > > >>>>It is extremely common that the available data set is too small to > > >partition the > > > >>>>data into in- and out- of sample sets of adequate size. In financial > > > >>>>research, the data set size is usually limited not by the data > >availability > > >but > > > >>>>by the data stationarity. To create valid sample sizes from small > > > >>>>data, > a > > > >>>>technique called "leave-one-out" or "bootstrapping" or "jackknifing" > > > >>>>is > > >used. In > > > >>>>those techniques the model is developed on the entire data except for > one > > > >>>>"holdout" point, then tested on this point. Then a different point is > > >selected > > > >>>>and the process is repeated. The validation results are obtained by > > >combining > > > >>>>the results of holdout points. Walk-forward optimization is an example > of > > >this > > > >>>>technique and actually reduces standard error in the more > > > >>>>important "out-of-sample" test. > > > > >>>>>better model would be the one which not only > > > >>>>>accounts for the supply/demand, but also for its changing elasticity > > > >>>>>over time > > > > >>>>That is definitely so and is often driven by seasonality as well > >as regime > > > >>>>shifts. For futures, such as ES, the elasticity could drift in > > > >>>>response > >to > > > >>>>the proximity of the expiration date or as a result of changing market > > >sentiment > > > >>>>or increased > > ... > > Erfahren Sie mehr » -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.
