Ken, For about a year now, I have been running a nightly exploration over all symbols of AMEX, NASDAQ, NYSE combined. The script itself applies logic to reject low liquidity stock, etc. and is applied to all stocks, rather than preprocessing a watchlist. The script is currently under 100 lines and completes in around 10 minutes (don't remember exactly, I stopped paying attention a long time ago).
Optimizing that same script over 108 iterations takes anywhere from 9 - 13 hours depending on the machine/memory (strictly low end equiptment - Pentium 4's). Doing a walk forward of any meaningful length would take days, except that I split the load over multiple machines. In an earlier incarnation, I had been writing out to composite symbols in order to retain a sorted rank of PositionScore based on all symbols scanned, not just ones that resulted in trade signals. I then made reference to those rankings within my custom backtest code to filter out signals that would not have occured due to no limit order actually having been placed (http://finance.groups.yahoo.com/group/amibroker/message/114739). Ultimately, that proved to be prohibitively expensive (at least as implemented, Thomasz had an alternate approach that was much faster at the cost of altering the strategy logic), taking over 30 minutes for an exploration. Optimizations were frustrating and impractical, leading to my eventual purchase of many cheap low end machines to farm out the load. Walk forward would have been out of the question. Fortunately, I was able to change the money management aspect of the system and move away from fixed number of positions. It's been trading profitably ever since :) Building on Herman's excellent suggestions, I would add that low end machines (1.6 Ghz) can be had for under $150 these days (I got lucky on eBay, paying $65 each for 7 of them!). Having a small farm allows you to spread the load of a single walk forward (i.e. each machine handling a different time period) or to backtest/optimize multiple different versions of your strategy at the same time. Mike --- In [email protected], "Ken Close" <[EMAIL PROTECTED]> wrote: > > A side question is this: what is the largest practical AFL file ever created > that runs and is used regularly? > > Obviously a hard question to answer (what is yours in terms of number of > lines or number of symbols operated on). We talk about memory and multiple > cores, and speed, etc, I will bet primarily in terms of optimization speed. > But what about just a large number of symbols and a large number of > calculations and variables? That is what my question is aimed at. > > If you are wondering why this question, here is a brief background: all of > my recent postings have been because I am trying to work my way towards > implementation of what seems like a very large system. I want to calculate, > for each and every symbol, a ranking indicator or value really. I want to > use the "ordinal value" approach that I have recently been asking about. > Creating these ordinal values consumes a LOT of processing time and > calculations etc. I would like to do this on 1000s of symbols (maybe the > entire database) but may have to settle for some subset, perhaps only in the > hundreds. But that is only the first step. > > After having these calculated values, I thought I would save them into the > OI field of each symbol. Then, separately, I would create an elaborate > rotational trading system, with a variety of parameters, and which would use > the ordinal ranking parameter I had previously calculated and saved into the > OI field. The rotational trading system, I am estimating, would take a lot > of statements and need to keep track of a lot of information, and that would > be before attempting any opimization of the trading parameters. I am > guessing that the rawbacktester mode is required. > > If separating the ranking calculation and saving it into the OI field is a > problem (because of the interplay between local storage vs plugin access > that I asked about in another message), then I could consider putting > everything (I mean everything) into one humongous AFL file. Calculate > ordinal rankings, keep them in their variables (by symbol) and then go right > into a rotational trading routine, all in the same code file. I suspect it > would either overload memory, or would take days of continual running to > complete. > > That is why I asked what is the largest code file you have ever made. Maybe > my quest is a futile one, but I am still plowing along. > > Thanks for listening. > > Ken >
