I'd prefer to handle this by doing the comparisons on a real portfolio of tradeables that has been driven only by market forces. But then again, that's what makes a market. Appreciate your thoughts.
--- In [email protected], "vlanschot" <[EMAIL PROTECTED]> wrote: > > My final comments. It has been illuminating. > > --- In [email protected], "quanttrader714" > <quanttrader714@> wrote: > > > > Good example, let's use it: "*random* trades on the S&P500 from > 1997 > > to 1999 will show an average expected return >zero." I agree. > Let's > > assume this period is OOS for your system and apply the test I > posted. > > To see if your entry method provides value added *when trading the > > S&P500 from 1997 to 1999*, it must do better than 95% (or whatever > > significance level you choose) of the backtests with random entries > > *on the S&P500 from 1997 to 1999*. Using simulated or any other > data > > would be comparing apples and oranges. Also, this example makes it > > easier to see why the number of trades should be around the same > as in > > the OOS backtest. > > I understand all that. Basically you compare a system's performance > with a multitude of alternative random systems. The expected > (average) outcome however of those random systems does not change > due to the same underlying series (i.e. history). This is different > from comparing a system's performance within a multitude of > alternative histories. Whatever you feel comfortable with. > > > > > Don't want to get into semantics either but no, > > I didn't mean the bootstrap > > > which is what I'd call resampling with replacement. You > > asked what I'd simulate if not market data. Well, in addition to > > random entries and exits, one could > > bootstrap (???) > > > a series of OOS trades > > and get a distribution of drawdows... or estimate the probability > of > > profit in n trades... or the probability of a string or n losing > > trades in a row... or... > > These are all valid and useful, particularly if they > are "bootstrapped" from multiple series of trades generated by a > system that was backtested against multiple histories. > > Thanks for the discussion. > > PS > > > > > --- In [email protected], "vlanschot" <ecbu@> wrote: > > > > > > --- In [email protected], "quanttrader714" > > > <quanttrader714@> wrote: > > > > > > > > I stressed OOS only because if you have enough trades, this > test > > > will > > > > work even with deliberately curve-fit systems. > > > > > > > > If you don't keep the number of trades approximately the same, > the > > > > comparisons won't be valid because some metrics are more > affected by > > > > the # trades than others. So you need to replicate that > aspect of > > > the > > > > OOS test but you're *randomly* doing it (drawing similarly > sized > > > > samples for an apples-to-apples comparison). So no, I don't > see any > > > > bias. > > > > > > The underlying series on which this is *randomly* done is > crucial, in > > > this case the same, e.g. 10,000 simulations of your series of > > > *random* trades on the S&P500 from 1997 to 1999 will show an > average > > > expected return >zero. > > > > > > > Monte Carlo simulations use that kind of input all the time. > > > > Which, BTW, this test is a form of. > > > You mean bootstrap. > > > > > > > > I wouldn't recommend simulating market data unless you can > somehow > > > > capture all the nuances, characteristics and > interrelationships that > > > > result from fear and greed and everything else that goes into > the > > > mix. > > > > > > Without getting too much into the semantics of things, but that > is > > > already captured in the market data? (Forget private info). If > not, > > > you simulate what? > > > > > > > > --- In [email protected], "vlanschot" <ecbu@> wrote: > > > > > > > > > > quanttrader714, > > > > > > > > > > Q for you: > > > > > > > > > > Not knowing the other settings, let's assume the system > shows > > > > > promising results over the IS-period (otherwise why bother > > > testing > > > > > further). Let's further assume that the risk/return profile > (s) of > > > the > > > > > underlying series is fairly stable over time. Is there not > > > already a > > > > > natural bias in the fact that the number of trades, > regardless of > > > IS > > > > > or OOS, is inticately linked to the aforementioned profile, > i.e. > > > the > > > > > expected return, simply because we assume "1 history"? > Therefore, > > > > > having buy-signals drawn "randomly" but benchmarked to the > number > > > of > > > > > trades in the OOS-period doesn't give you an unbiased view > of the > > > > > system versus chance? > > > > > > > > > > FAC, I'm not criticising you. I realize your suggestion is > meant > > > as a > > > > > quick test, but I would suggest to extend it via MCS: > generate > > > > > simulated price-series (stress-tested or not), thus > generating > > > > > hundreds of "alternative histories" and apply one's system > to > > > these. > > > > > All this can already be achieved in AB now, although TJ is > > > planning a > > > > > native MCS-functionality. > > > > > > > > > > PS > > > > > > > > > > (For Brian: unfortunately Capra hates the markets [see his > > > > > book "Hidden Connections"]. Tried to explain things to him. > He > > > didn't > > > > > want to listen. Suggest private e-mail if you want to know > more). > > > > > > > > > > --- In [email protected], "quanttrader714" > > > > > <quanttrader714@> wrote: > > > > > > > > > > > > This is OT on psychology but a while back I believe you > were > > > asking > > > > > > about statistics and trading? Here's a very simple > statistical > > > test > > > > > > that can be run using AB alone. This simplified example > will > > > > > estimate > > > > > > the strength of a "long only" system's entries. Long and > short > > > > > > systems and exits are a bit trickier but the principle is > the > > > same. > > > > > > > > > > > > Run an *out of sample* (OOS) system backtest. Save the > > > results. > > > > > Note: > > > > > > OOS only! > > > > > > > > > > > > Add the following line of code to specify the number of > > > iterations. > > > > > > I'd run 1000 or more but as few as 100 will still give a > crude > > > > > > estimate. > > > > > > > > > > > > Iterations = Optimize("Iteration",1,1,1000,1); > > > > > > > > > > > > Replace the system's buy condition with the following code > but > > > leave > > > > > > the original settings, sell condition and stops in place. > > > Tweak the > > > > > > value in the Buy line (0.975 in this case) so the number > of > > > trades > > > > > is > > > > > > approx. the same number as in the original OOS backtest. > BTW, I > > > > > > personally wouldn't be comfortable with this procedure > unless > > > the > > > > > OOS > > > > > > backtest has at least several hundred trades. > > > > > > > > > > > > Buy= Random()>0.975; > > > > > > > > > > > > Optimize over the OOS period. Sort results by the metric > you > > > want to > > > > > > analyze. The fraction of optimized results that is > greater > > > than or > > > > > > equal to the OOS backtest metric is an estimate of the > > > probability > > > > > > that one can do as well as or better than the original > system > > > entry > > > > > by > > > > > > chance alone. Of course no matter how good the results, > > > there's no > > > > > > guarantee of future profitability. But this is an easy > way to > > > get a > > > > > > decent estimate of how much better than chance your OOS > metrics > > > are. > > > > > > > > > > Please note that this group is for discussion between users only. To get support from AmiBroker please send an e-mail directly to SUPPORT {at} amibroker.com For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: http://www.amibroker.com/devlog/ For other support material please check also: http://www.amibroker.com/support.html Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/amibroker/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/amibroker/join (Yahoo! ID required) <*> To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
