> > I have been reading through the archived posts on the raw data format. > The general conclusion seems to have been that raw data (i.e. the full > order book) is nice but impractical because there are ~2 million rows > per day per contract and so backtesting would take much too long. > > This may be true if the data is stored as ASCII-CSV in an ordinary > disk file, but has anyone looked into column-oriented DBMS such as > MonetDB? It is specifically designed for that type of problem. > > http://en.wikipedia.org/wiki/Column-oriented_DBMS > http://en.wikipedia.org/wiki/MonetDB > http://monetdb.cwi.nl/ > http://monetdb.cwi.nl/projects/monetdb/SQL/Benchmark/TPCH/index.html > > I haven't tried it. I suspect it might be possible to do JBT-type > stuff with full order books (raw data) on a reasonable modern system > with MonetDB or one of the similar column-oriented databases. Any > thoughts? > >
Even if the database access is instantaneous, it would still be prohibitively slow to process the raw limit book on every change (which happens about 30 times per second, on average). In all likelihood, your strategy would perform some sort of filtering/smoothing/sampling of the raw book data before making a trading decision. So ultimately, it's not the raw data that would be used, but some projection of it. This is what JBT does, it stores the projection in place of the raw data. It's debatable which projection is optimal. JBT uses 1-second snapshots of the book which average the book changes in the last second and store the result as a "book balance". I've experimented with other projections, as well, but the current one proved to be the most stable and conductive to profitability. This doesn't mean, of course, that there are not better projections, and if anyone has found one, feel free to share with the group. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en -~----------~----~----~----~------~----~------~--~---
