I stressed OOS only because if you have enough trades, this test will work even with deliberately curve-fit systems.
If you don't keep the number of trades approximately the same, the comparisons won't be valid because some metrics are more affected by the # trades than others. So you need to replicate that aspect of the OOS test but you're *randomly* doing it (drawing similarly sized samples for an apples-to-apples comparison). So no, I don't see any bias. Monte Carlo simulations use that kind of input all the time. Which, BTW, this test is a form of. I wouldn't recommend simulating market data unless you can somehow capture all the nuances, characteristics and interrelationships that result from fear and greed and everything else that goes into the mix. --- In [email protected], "vlanschot" <[EMAIL PROTECTED]> wrote: > > quanttrader714, > > Q for you: > > Not knowing the other settings, let's assume the system shows > promising results over the IS-period (otherwise why bother testing > further). Let's further assume that the risk/return profile(s) of the > underlying series is fairly stable over time. Is there not already a > natural bias in the fact that the number of trades, regardless of IS > or OOS, is inticately linked to the aforementioned profile, i.e. the > expected return, simply because we assume "1 history"? Therefore, > having buy-signals drawn "randomly" but benchmarked to the number of > trades in the OOS-period doesn't give you an unbiased view of the > system versus chance? > > FAC, I'm not criticising you. I realize your suggestion is meant as a > quick test, but I would suggest to extend it via MCS: generate > simulated price-series (stress-tested or not), thus generating > hundreds of "alternative histories" and apply one's system to these. > All this can already be achieved in AB now, although TJ is planning a > native MCS-functionality. > > PS > > (For Brian: unfortunately Capra hates the markets [see his > book "Hidden Connections"]. Tried to explain things to him. He didn't > want to listen. Suggest private e-mail if you want to know more). > > --- In [email protected], "quanttrader714" > <quanttrader714@> wrote: > > > > This is OT on psychology but a while back I believe you were asking > > about statistics and trading? Here's a very simple statistical test > > that can be run using AB alone. This simplified example will > estimate > > the strength of a "long only" system's entries. Long and short > > systems and exits are a bit trickier but the principle is the same. > > > > Run an *out of sample* (OOS) system backtest. Save the results. > Note: > > OOS only! > > > > Add the following line of code to specify the number of iterations. > > I'd run 1000 or more but as few as 100 will still give a crude > > estimate. > > > > Iterations = Optimize("Iteration",1,1,1000,1); > > > > Replace the system's buy condition with the following code but leave > > the original settings, sell condition and stops in place. Tweak the > > value in the Buy line (0.975 in this case) so the number of trades > is > > approx. the same number as in the original OOS backtest. BTW, I > > personally wouldn't be comfortable with this procedure unless the > OOS > > backtest has at least several hundred trades. > > > > Buy= Random()>0.975; > > > > Optimize over the OOS period. Sort results by the metric you want to > > analyze. The fraction of optimized results that is greater than or > > equal to the OOS backtest metric is an estimate of the probability > > that one can do as well as or better than the original system entry > by > > chance alone. Of course no matter how good the results, there's no > > guarantee of future profitability. But this is an easy way to get a > > decent estimate of how much better than chance your OOS metrics are. Please note that this group is for discussion between users only. To get support from AmiBroker please send an e-mail directly to SUPPORT {at} amibroker.com For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: http://www.amibroker.com/devlog/ For other support material please check also: http://www.amibroker.com/support.html Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/amibroker/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/amibroker/join (Yahoo! ID required) <*> To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
