As far as I know, nothing can compare to AB in terms of handling huge amount of data and huge number of trades, not even close!
On Thu, Sep 24, 2009 at 10:34 AM, Herman <[email protected]> wrote: > > > Ryan, > > > I cannot add anything to your report, except that I greatly enjoyed reading > it. Thank you for sharing your results! > > > Of course I was happy to learn that AmiBroker did well. Congratulations on > this one Tomasz ;-) > > > best regards, > > herman > > > > Thursday, September 24, 2009, 9:55:37 AM, you wrote: > > > > Hello All, > > > > I have just completed a few days of testing several modeling/backtesting > > > programs. I thought, perhaps, the other members of the list might find > the > > > results useful; as I am new here, hopefully this can serve as my first > productive contribution. > > > > I do a lot of testing & modeling on (a) daily bars (looking to execute > > > intraday trades) and (b) tick data (for short-term trades). In the case > of the > > > first, I need to create signals on a daily series but execute orders > against a > > > 1M intraday series. So, in order to test an idea over five years of data, > I > > > have to do it over 5Y of 1M data. This gets into performance issues when > you > > > start wanting to run a test on several issues and want to run multiple > > > revisions of the test. As for testing on tick data, I am sure you are > familiar > > > (at least conceptually) with the performance issues there. > > > > I took a look at NinjaTrader, TradeStation, OpenQuant & AmiBroker. I > created > > > an ASCII file of 5Y of EURUSD, USDJPY, GBPUSD, AUDUSD & USDCAD 1M data. > In > > > each program, I ran a simple EMA crossover test (10/50). It was an > obnoxious > > > test, resulting in +300,000 trades, but it was easy to implement and was > a > > > good stress test. What I wanted to do was see if I could get: a) reliable > > > (e.g. reproducible) results from a single-security test and b) a test of > the > > > five FX pairs as a portfolio (again, in a reliable/reproducible manner). > > > > 1. Ninja: I use Ninja daily to scalp with and execute some short-term > system > > > code. I had dim hopes for the backtesting since I am familiar with the > program > > > - Ninja really is an execution platform first and an analytical platform > > > second. The results were more or less what I expected: I could get it to > test > > > one issue with reproducible results at an okay speed (about 3:00M) but it > > > would start to go into fits when I ran it against all five at once. The > > > results of the five issue test would vary from instance to instance - it > would > > > usually show the results for the first 3 issues correctly, but on the > last 2 > > > it would suffer some kind of memory issue and give me numbers that were > > > totally off. In one pass, it even managed to corrupt itself and I had to > reload all the data. > > > > 2. TradeStation: I have a lot of time invested in TradeStation and I was > > > already familar with it's problems - mainly, that over a large test set, > TS > > > will return different test results. I have talked with TS support and > posted > > > on the message board about this, but I never got anyone interested in > what I > > > found to be a critical issue. The results of this test were as expected: > I > > > could not get two results to match. Any time I would refresh data from > the TS > > > data servers and run the test again, I would get a different result. > Sometimes > > > it was as much as test 1 being -$190K and test 2 being +$74K. I do not > > > understand how anyone can use this tool for successful modeling if they > are > > > testing over a large dataset; just making up a number would have been as > > > useful. I even exported and imported the data to ensure that it wasn't an > > > issue with the TS data servers. Same inconsistency. I couldn't test all > five > > > pairs together since TS does not do portfolio backtesting. As > > > for time of a single test, it is hard to tell with TS as to what, > exactly, > > > it is waiting for/trying to do at any given moment, but I would say it > would > > > usually take about 2/4M per test (although, I have no idea what it was > doing > > > in that 2/4M since the results it returned seemed random at times). > > > > 3. OpenQuant: This platform looked interesting. I set it up, imported the > > > data and ran a test. After 20M I noticed that the first EURUSD 1M test > was at > > > 10%. I closed it down and uninstalled it. The performance was just not > going to work for me. > > > > 4. AmiBroker: You can likely guess the results as I am here as a new > member. > > > AB was able to test EURUSD in about 30 seconds and was able to do the > > > portfolio of all 5 pairs in about 2M. No matter how many times I ran the > test, > > > the numbers it would return were the same. This fact is pretty crucial to > my > > > work. I understand that data will have errors in it, but I at least need > my > > > modeling software to consistently return results (which AB did). I even > > > created new databases and populated them with the same data - all test > results returned were the same. > > > > As you all know, AB was, by far, the cheapest option out of all of the > above. > > > > I hope some members find these results useful/interesting and I think > that > > > these results really are a credit to Tomasz. > > > > I would be > > > interested to hear from anyone who has tested AB against any other > > > off-the-shelf tools that I did not look at. > > > > - Tim > > > > > > > > > > > > ------------------------------------ > > > > **** IMPORTANT PLEASE READ **** > > > This group is for the discussion between users only. > > > This is *NOT* technical support channel. > > > > TO GET TECHNICAL SUPPORT send an e-mail directly to > > > SUPPORT {at} amibroker.com > > > > TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at > > > http://www.amibroker.com/feedback/ <http://www.amibroker.com/feedback/> > > > (submissions sent via other channels won't be considered) > > > > For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG: > > > http://www.amibroker.com/devlog/ <http://www.amibroker.com/devlog/> > > > > Yahoo! Groups Links > > > > http://groups.yahoo.com/group/amibroker/ > > > > Individual Email | Traditional > > > > http://groups.yahoo.com/group/amibroker/join > > > (Yahoo! ID required) > > > > > > mailto:[email protected]<[email protected]> > > > > > > mailto:[email protected]<[email protected]> > > > > [email protected] > > > > http://docs.yahoo.com/info/terms/ > > > > >
