I'm definitely aware that I can save the data however I want. I've modified the format extensively in the past, and continue to do so. Similarly, I know that the *entire* book is too much data for optimization jobs due to the sub-second updates, etc. I was hoping to spur a discussion of how others may have modified the format, and if any of those modifications might be considered candidates for inclusion into the baseline.
For instance, I have not touched the 1-second resolution, but for what I'm currently testing, I've added a single price field to each line that gives me additional insight into the state of the book. -Adam On Jul 8, 6:07 pm, nonlinear5 <[email protected]> wrote: > > As I understand it, the major reason for not saving the entire book > > was that the data files would get way too big way too quickly. Large > > data files would be difficult to share over the net, and would add a > > lot of disk overhead during optimization and backtesting. > > > Over the last couple of years, it seems to me that the amount of data > > file sharing has dropped off dramatically. Other aspects of the > > optimizer performance have been improved, which might now be able to > > be traded for disk usage. In my backtesting and optimization system, > > I've got a 7-disk 3TB ZFS Raid, a dual-processor multi-core CPU > > configuration, and loads more RAM than I had years ago. I'm also on > > fiber now, instead of cable. Data file size is really not the same > > obstacle it used to be for me. > > Disk space is cheap, indeed. It was never the issue, however. The > problem is with backtesting and optimizing these enormous quantities > of data. If you want to capture the entire book, it would be 40 items: > 10 bid prices, 10 bid sizes, 10 ask prices, and 10 ask sizes. Each of > these 40 items changes approximately 4 times per second. This totals > to about 40 * 4 = 160 data pieces per second. This mean that JBT would > take about 100 times longer to backtest and optimize. I have a pretty > fast machine, but some optimization jobs take several hours. With the > "full book capture", these jobs would take weeks to complete. -- You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en.
