Paul, Initially I had been doing it manually. It hasn't been that bad (e.g. copying and pasting the html output of individual walk forward results into single unified file).
However, I have very recently begun experimenting with scripts to automate the process and stitch the results back togeather. I have a few ideas that I'm exploring, but am not yet able to discuss 'till I've had a chance to further organize my thinking and do a proof of concept. It is still early in my efforts, and I have limited time (nights after work and once the family is in bed), but first results look promising. In any event, I will gladly make available whatever I come up with once it's reached a minimal level of functionality. So, if your spare equiptment is not causing you too much hardship, it may be worth hanging on to it a little longer. Mike --- In [email protected], "Paul Ho" <[EMAIL PROTECTED]> wrote: > > Mike, > Are you spreading your load over several machines manually, or you're using > programs/scripts to automate that process? > I have a few old machines that I was thinking about throwing out, may be I > could utilise them if there was a practical way to share the load. > /Paul. > > > _____ > > From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf > Of Mike > Sent: Wednesday, 9 July 2008 6:26 PM > To: [email protected] > Subject: [amibroker] Re: What is your Largest AFL file? > > > > Ken, > > For about a year now, I have been running a nightly exploration over > all symbols of AMEX, NASDAQ, NYSE combined. The script itself applies > logic to reject low liquidity stock, etc. and is applied to all > stocks, rather than preprocessing a watchlist. The script is > currently under 100 lines and completes in around 10 minutes (don't > remember exactly, I stopped paying attention a long time ago). > > Optimizing that same script over 108 iterations takes anywhere from > 9 - 13 hours depending on the machine/memory (strictly low end > equiptment - Pentium 4's). Doing a walk forward of any meaningful > length would take days, except that I split the load over multiple > machines. > > In an earlier incarnation, I had been writing out to composite > symbols in order to retain a sorted rank of PositionScore based on > all symbols scanned, not just ones that resulted in trade signals. I > then made reference to those rankings within my custom backtest code > to filter out signals that would not have occured due to no limit > order actually having been placed > (http://finance. > <http://finance.groups.yahoo.com/group/amibroker/message/114739> > groups.yahoo.com/group/amibroker/message/114739). > > Ultimately, that proved to be prohibitively expensive (at least as > implemented, Thomasz had an alternate approach that was much faster > at the cost of altering the strategy logic), taking over 30 minutes > for an exploration. Optimizations were frustrating and impractical, > leading to my eventual purchase of many cheap low end machines to > farm out the load. Walk forward would have been out of the question. > > Fortunately, I was able to change the money management aspect of the > system and move away from fixed number of positions. It's been > trading profitably ever since :) > > Building on Herman's excellent suggestions, I would add that low end > machines (1.6 Ghz) can be had for under $150 these days (I got lucky > on eBay, paying $65 each for 7 of them!). Having a small farm allows > you to spread the load of a single walk forward (i.e. each machine > handling a different time period) or to backtest/optimize multiple > different versions of your strategy at the same time. > > Mike > > --- In [EMAIL PROTECTED] <mailto:amibroker%40yahoogroups.com> ps.com, "Ken > Close" <ken45140@> wrote: > > > > A side question is this: what is the largest practical AFL file > ever created > > that runs and is used regularly? > > > > Obviously a hard question to answer (what is yours in terms of > number of > > lines or number of symbols operated on). We talk about memory and > multiple > > cores, and speed, etc, I will bet primarily in terms of > optimization speed. > > But what about just a large number of symbols and a large number of > > calculations and variables? That is what my question is aimed at. > > > > If you are wondering why this question, here is a brief > background: all of > > my recent postings have been because I am trying to work my way > towards > > implementation of what seems like a very large system. I want to > calculate, > > for each and every symbol, a ranking indicator or value really. I > want to > > use the "ordinal value" approach that I have recently been asking > about. > > Creating these ordinal values consumes a LOT of processing time and > > calculations etc. I would like to do this on 1000s of symbols > (maybe the > > entire database) but may have to settle for some subset, perhaps > only in the > > hundreds. But that is only the first step. > > > > After having these calculated values, I thought I would save them > into the > > OI field of each symbol. Then, separately, I would create an > elaborate > > rotational trading system, with a variety of parameters, and which > would use > > the ordinal ranking parameter I had previously calculated and saved > into the > > OI field. The rotational trading system, I am estimating, would > take a lot > > of statements and need to keep track of a lot of information, and > that would > > be before attempting any opimization of the trading parameters. I > am > > guessing that the rawbacktester mode is required. > > > > If separating the ranking calculation and saving it into the OI > field is a > > problem (because of the interplay between local storage vs plugin > access > > that I asked about in another message), then I could consider > putting > > everything (I mean everything) into one humongous AFL file. > Calculate > > ordinal rankings, keep them in their variables (by symbol) and then > go right > > into a rotational trading routine, all in the same code file. I > suspect it > > would either overload memory, or would take days of continual > running to > > complete. > > > > That is why I asked what is the largest code file you have ever > made. Maybe > > my quest is a futile one, but I am still plowing along. > > > > Thanks for listening. > > > > Ken > > >
