I was playing with adding the bid/ask size. I know in stocks sometimes when things heat up you see a spike in the size of one of those (but the rest of the book kind-of stands still). I didn't play with it too much though.
On Thu, Jul 8, 2010 at 8:51 PM, wigginx <[email protected]> wrote: > I'm definitely aware that I can save the data however I want. I've > modified the format extensively in the past, and continue to do so. > Similarly, I know that the *entire* book is too much data for > optimization jobs due to the sub-second updates, etc. I was hoping to > spur a discussion of how others may have modified the format, and if > any of those modifications might be considered candidates for > inclusion into the baseline. > > For instance, I have not touched the 1-second resolution, but for what > I'm currently testing, I've added a single price field to each line > that gives me additional insight into the state of the book. > > -Adam > > On Jul 8, 6:07 pm, nonlinear5 <[email protected]> wrote: > > > As I understand it, the major reason for not saving the entire book > > > was that the data files would get way too big way too quickly. Large > > > data files would be difficult to share over the net, and would add a > > > lot of disk overhead during optimization and backtesting. > > > > > Over the last couple of years, it seems to me that the amount of data > > > file sharing has dropped off dramatically. Other aspects of the > > > optimizer performance have been improved, which might now be able to > > > be traded for disk usage. In my backtesting and optimization system, > > > I've got a 7-disk 3TB ZFS Raid, a dual-processor multi-core CPU > > > configuration, and loads more RAM than I had years ago. I'm also on > > > fiber now, instead of cable. Data file size is really not the same > > > obstacle it used to be for me. > > > > Disk space is cheap, indeed. It was never the issue, however. The > > problem is with backtesting and optimizing these enormous quantities > > of data. If you want to capture the entire book, it would be 40 items: > > 10 bid prices, 10 bid sizes, 10 ask prices, and 10 ask sizes. Each of > > these 40 items changes approximately 4 times per second. This totals > > to about 40 * 4 = 160 data pieces per second. This mean that JBT would > > take about 100 times longer to backtest and optimize. I have a pretty > > fast machine, but some optimization jobs take several hours. With the > > "full book capture", these jobs would take weeks to complete. > > -- > You received this message because you are subscribed to the Google Groups > "JBookTrader" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<jbooktrader%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/jbooktrader?hl=en. > > -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.
