Hi, Everyone, Thanks for all the input on the memory usage issues. There is a updated version of FastBit source code marked as ibis1.2.5 with some initial implementation of aggregation operations by parts. This should reduce the memory usage for aggregation operations that does not require all the data to be in memory.
The new version is available at http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 We have also included two patches from Jon Strabala and Robert Wong to address a couple of different issues. This release also fixes a number issues raised by Alexandre Maure, Anderson Carniel, Jon Strabal, Andrew Olson, Teryl Taylor, Robert Wong, and others. Please feel free to let us know if you encounter any problems with the new source code. John PS: Just in case you are wondering about which aggregations can be performed by parts. The aggregation operators, count, sum, min, and max can be performed by parts. The operators, avg, var, and stddev can be transformed into sums and counts and are currently internally transformed into sums and counts. This transformation may introduce very slightly different results. The computations are done with double-precision floating-point numbers, which have 16-digit precision. The difference should be limited to the last couple of digits and should not be noticeable for most users. However, those users with very sensitive computations should know that the converted expression have different round-off properties from the original expression. The bad operations are distinct and median, both which require all data to be in memory. PPS: We have made some progress in supporting joins through a new class called quaere. We intend to eventually rename it to ibis::query. _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
