Hi, Michael, Thanks for the update.
If your 100 records were scattered around in the original data set, collecting them together in memory is a reasonable approach. However, it might not be worthwhile to index them. On the other hand, if you access the data is some particular order, it might be useful to order the data accordingly. It would be relatively quick to order the 100 rows. Hope this is useful. John On 4/7/11 12:24 PM, Chong, Michael wrote: > Hi John, > > Just to follow up on this story. I tried to put in the indexes pre-load as > you suggested and I think they helped a bit. However what seemed to help a > lot was to select bigger chunks of data. > > I have perhaps 10,000 events to sort through. Each event has a subset of > data. I would pick event X and then do the calculations. Processing each > event means hitting the disk to grab data. > > What I do now instead is to pick say 100 events, so grab 100 X worth of data > in a big select, and use that data to build a smaller table in memory and > finally add a fast bit index using buildIndex to the in memory table. Then > using another loop look through that table 100 times to do my original > calculation. This seems to help a lot and I am seeing perhaps a factor of 10x > speedup. > > I guess it is better to gulp your data than sip it. Some params are: > > Umbutu linux, 24 GB RAM, 4 x 6 Core machine. But for this exercise I am using > one core and single threaded. > > Thanks again for your kind help John. > > Warmest regards, Mike. > > DISCLAIMER: This e-mail message and any attachments are intended solely for > the use of the individual or entity to which it is addressed and may contain > information that is confidential or legally privileged. If you are not the > intended recipient, you are hereby notified that any dissemination, > distribution, copying or other use of this message or its attachments is > strictly prohibited. If you have received this message in error, please > notify the sender immediately and permanently delete this message and any > attachments. > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
