Hello Fred, Precisely.
I'm not going crazy after-all! What method or methods will tell us how the system is likely to perform out of sample; since in the end system trading is nothing but a perpetual walk forward test? BrianB2. --- In [email protected], "Fred" <[EMAIL PROTECTED]> wrote: > > While MCS is a good tool for validating some things it is not a > substitute for out of sample and/or walk forward testing ... If for > example I: > > - Write a system > - Test it to make sure the rules are working as intended > - Optimize the variables to have the system produce the best results > it can based on some metric or metrics within the confines of the > rules > - Use an MCS on the trades that are generated > > This tells me nothing about how the system is likely to perform out > of sample. It only tells me about the statistics related to the > optimized rules of the system which are the result of scrambling the > order of the trades that resulted from using the same in sample data > that system was optimized on. > > --- In [email protected], "sebastiandanconia" > <sebastiandanconia@> wrote: > > > > I only offer this as a consideration when using such testing, not > as a > > criticism of Monte Carlo Simulations. A subtle but significant > point > > (IMO) when using MCS: They may or may not be applicable to > > trading/investing, because the markets don't always behave > randomly. > > > > An example of when a MCS would clearly be appropriate: Let's say > you're > > a defense contractor manufacturing a part for the International > Space > > Station. The part is critical, but because of limitations in > > engineering technology it has a high failure rate, and there's no > way of > > forecasting in advance if a part will fail. However, although the > > failure rate is high, it's also very consistent. Until technology > > advances sufficiently the only practical solution is to keep > plenty of > > spares on-hand. > > > > A MCS could tell you what the optimal number of spares to keep on- > hand > > would be. The part failures are random, but MC could tell you the > > likelihood of two, three, five, ten, etc., consecutive failures. > You > > might determine that there would only be a 1/100,000 chance that 6 > > spares in a row would fail, so you might advise NASA to stock at > least 6 > > spares at all times. > > > > Some trading systems, though, will be successful because they take > > advantage of repeating sequences of events, not random events. > Business > > cycles go through a specific sequence, company growth follows a > certain > > pattern from infancy to maturity, price trends/reversals follow a > > sequence, etc. If trades based on reliable, repeating patterns are > > taken out of order by a MCS such that a massive drawdown or a > > bankrupting series of losers occurs, that can distort the value of > the > > trading method by putting the trades in an order that wouldn't > occur. > > > > Soapbox alert!:) Another reason that "Why does it work?" is such > an > > important question with trading systems, since a good answer to > that > > question can lead to a good answer for another important > question, "When > > WON'T it work?" > > > > > > S. > > > > > > > > --- In [email protected], "brian.z123" <brian.z123@> wrote: > > > > > > OT:margin of error example. > > > > > > As the trader is more interested in the general population of > future > > > trades than the test sample, what can be learnt from the sample? > > > > > > One answer is to trade the system for a decade or two and find > out. > > > Another option is to simulate decades or even centuries of > trading > > > by applying Monte Carlo analysis. > > > In laymans terms MCS is a computer generated, random walk through > > > *all*, of the possible trading outcomes based on the trading > sample > > > provided. > > > The result is a report or system profile that provides > statistics on > > > which to base our levels of trading confidence for the future. > > > There are other ways of sneaking a peak into a trading systems > > > future but MCS is the most commonly used. > > > I have developed my own system, that I don¡¦t want to headline > here > > > for various reasons, not the lest of which is that I can¡¦t > provide > > a > > > mathematical proof if called on to do so. > > > > > > Assuming that an MCS has been conducted on a sample of 50 trades > > > produced by a back-tested system and the report indicates that > the > > > meanW/meanL for the system over a large number of trading > > > simulations is 53/47. The StDev is 40% for both Wins and Losses. > > > How confident can we be in that result? > > > > > > From David Lanes statistical website: > > > http://davidmlane.com/hyperstat/A103397.html > > > The standard error of a statistic is the standard deviation of > the > > > sampling distribution of that statistic. > > > The formula for the standard error of the mean is: > > > > > > StErrorOfMeanPopulation = StDevPopulation/SqRt(N)Sample > > > > > > For any statistic: > > > > > > StErrorOfMeanPopulation(statistic) = StDevPopulation > (statistic)/SqRt > > > (N)Sample > > > > > > Applying the StdErrorMean equation to the example: > > > > > > Back-test sample size N = = 50, > > > MCS meanWin/meanLoss = = 53/47, > > > MCS Win StDev% = = 40%, > > > MCS Win StDev$ = = 40% x 53 = = 21.2, > > > MCS Loss StDev% = = 40%, > > > MCS Loss StDev$ = = 40% x 47 = = 18.8, > > > > > > StdError%Wins = = 40/SqRt(50) = = multiply mean by +/- 5.6 %, > > > Trading Win range = = 50 ¡V 56, > > > (min = = 53 x 0.943 = = 50, max = = 53 X 1.056 = = 56). > > > > > > The same result can be obtained using StDev as a number ($) > rather > > > than as a percentage. > > > > > > StdError$Wins = = 21.2/SqRt(50) = = +/- 3 = = Win range = = 47 > +/-3 > > > = = 50 -56 . > > > > > > Repeating the calculations for Losses shows the the mean Losses > > > range between 44 ¡V 50. > > > > > > I chose this extreme example to demonstrate the outcome for a > small > > > back-test sample with high volatility trades and a small win/loss > > > margin. > > > > > > If the same trading pattern were generated from a back-test > sample > > > of 2500 trades and the simulated meanWins and mean Losses each > had a > > > StDev of 10% the range for the margin of error would be: > > > > > > Wins 52.9 ¡V 53.1, > > > Losses 46.9 ¡V 47.1. > > > > > > This means that the we can be 95% confident the real mean values > are > > > somewhere within those ranges. > > > For a higher level of confidence the range will be greater. > > > > > > Resorting to the age-old teaching trick of asking the students > for > > > the answer while pretending to already know it yourself; can > anyone > > > in the forum tell me if this is the correct way to use StdError > when > > > applied to trading? > > > > > > > > > > > > BrianB2 º > > > --- In [email protected], "brian.z123" brian.z123@ > > > wrote: > > > > > > > > Part1 of Project Based Training No1. > > > > > > > > The objective of the project is to introduce new traders to the > > > main > > > > concepts of system design/testing and demonstrate their > > > application > > > > in AmiBroker. > > > > At the same time it is hoped that the ideas presented will > provoke > > > > discussion and provide trading stimulation. > > > > > > > > All of the stages in the design process will not be > demonstrated > > > as > > > > most have already been covered elsewhere in the AmiBroker > support > > > > material. > > > > > > > > A basic understanding of the application of some statistical > > > methods > > > > to the trading environment is a pre-requisite. > > > > The opening topics address this need. > > > > > > > > To those who find the subject matter new *the project* will be > a > > > > workbook . > > > > To those who have experience in the subject it will be an > > > > opportunity to workshop. > > > > > > > > I would like to acknowledge my indebtedness to the academic > > > > community . > > > > I often refer to the material so generously interpreted for the > > > > layperson and made available at websites by academic > specialists, > > > > particularly those associated with Universities. > > > > > > > > > ******************************************************************* > > > > Margin of Error. > > > > > > > > Back-testing of historical data provides traders with a > sample, > > > > typical of the trade they are testing. From that sample they > make > > > > inferences about the larger group, or population, of all past > > > trades > > > > and future trades, of the same type, that were not included in > > > their > > > > test window. > > > > Despite the fact that the people who teach them to back-test > also > > > > teach them that the past can not predict the future, some > continue > > > > to act as if it can. > > > > > > > > If the past can't predict the future. How can anyone trade with > > > > confidence? > > > > > > > > The answer is that while the future can't be predicted, the > > > > likelihood of some mathematically defined outcomes can be > > > predicted > > > > with a degree of confidence. > > > > Statistics is the mathematical discipline that manages that > very > > > > well. > > > > > > > > The caveat is that to apply statistical methods to trading > > > samples, > > > > the assumption is made that they are the result of a random > > > process. > > > > Where the trading system chosen is biased to non-random > behaviour > > > it > > > > will be prone to failure if the market acts contrary to that > bias. > > > > > > > > For that reason system traders are faced with a choice between > > > > attempting to define market behaviour e.g. a trend, and pick a > > > > system to suit that, or search for a universal signal that is > > > > consistent irrespective of any assumed market bias. > > > > > > > > If statistics can predict the likelihood of future trading > > > outcomes, > > > > how accurate will it be? > > > > > > > > *Standard error* or *margin of error* offers traders a > solution > > > but > > > > they are not subjects that are often discussed. > > > > > > > > In his book ,*Design, Testing, and Optimisation of Trading > > > Systems* > > > > (John Wiley & Sons, 1992), Robert Pardo raises the issue of the > > > > accuracy of trading *predictions* based on the size of the > sample > > > > used: > > > > > > > > * The sample size must be large enough to allow the trading > system > > > > to generate a statistically significant sample of trades. > > > > A sample of one trade is certainly insignificant, whereas a > sample > > > > of 50 trades or more is generally adequate.* > > > > > > > > He uses Standard Error as a measure of significance: > > > > > > > > StdError = = 1/SquareRoot(sample size), > > > > > > > > 1/SqRt(50) = = 14.1%. > > > > > > > > There is little by way of further explanation provided. > > > > > > > > Applying the formula to a greater number of samples: > > > > > > > > Where N = = the number of trades in the sample > > > > > > > > StdError factor = = 1/SqRt(N) > > > > StdError% = 1/SqRt(N) * 100 > > > > > > > > If N = = 2500 the StdError% = = 1/SqRt(2500) * 100 = = +/- 2% > > > > If N = = 10000 the StdError% = = 1/SqRt(10000) * 100 = = +/- > 1% > > > > > > > > A trade sample of 10000 to provide statistical accuracy of 1% > is > > > not > > > > easily achievable for traders, although a lot easier than > > > accurately > > > > surveying the eye colour of Polar Bears. > > > > > > > > Pardos equation is in fact, a rounding of the StdError equation > > > for > > > > a 95% level of confidence: > > > > > > > > Margin of error at 99% confidence = = 1.29/SqRt(N) > > > > Margin of error at 95% confidence = = 0.98/SqRt(N) > > > > Margin of error at 90% confidence = = 0.82/SqRt(N) > > > > > > > > Later in the project I will use a basic random number > generator, > > > > within Xcel, to provide a visual aid that traders can use to > > > > understand the *sample* concept and decide for themselves what > > > > constitutes an adequate sample. > > > > > > > > Wikipedia provides some additional clarity on the subject: > > > > > > > > http://en.wikipedia.org/wiki/Margin_of_error > > > > > > > > *The margin of error expresses the amount of the random > variation > > > > underlying a survey's results. This can be thought of as a > measure > > > > of the variation one would see in reported percentages if the > same > > > > poll were taken multiple times. The larger the margin of error, > > > the > > > > less confidence one has that the poll's reported percentages > are > > > > close to the "true" percentages, that is the percentages in the > > > > whole population.* > > > > > > > > *An interesting mathematical fact is that the margin of error > > > > depends only on the sample size and not on the population size, > > > > provided that the population is significantly larger than the > > > sample > > > > size, and provided a simple random sample is used. Thus for > > > > instance¡K¡K.the running example with 1,013 random > > samples¡K¡Kwould > > > > yield essentially the same margin of error (4% with a 99% > level of > > > > confidence) regardless of whether the > > population¡K¡K¡K.consisted of > > > > 100,000 or 100,000,000.* > > > > > > > > In short the tail of the trading system sample is swinging the > > > > trading system cat. > > > > > > > > BrianB2 > > > > > > > > The material contained in this topic is for educational and > > > > discussion use only. > > > > It is not intended as financial advice and should not be > construed > > > > as such. > > > > The author is not an accredited academic or financial advisor. > > > > > > > > > > Please note that this group is for discussion between users only. To get support from AmiBroker please send an e-mail directly to SUPPORT {at} amibroker.com For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: http://www.amibroker.com/devlog/ For other support material please check also: http://www.amibroker.com/support.html Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/amibroker/ <*> Your email settings: Individual Email | Traditional <*> To change settings online go to: http://groups.yahoo.com/group/amibroker/join (Yahoo! ID required) <*> To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
