Hi, John Thank you for the quick response. The problems I reported are now gone and I'm able to aggregate much larger data thanks to the new feature!
There is one other thing, I've noticed. When aggregating, the number of different rows is determined by the first part, so if the first part will return only two rows, and second would give me 4, when used together I get only the two. Here are the commands to reproduce: This gives two rows: ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" This gives four rows: ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" This works on both parts and gives only two rows. ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" Please find the test data at http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz Petr PS.: I track the svn in git and I have made some changes so that I can easily create an rpm package from the library. Now I only have to run "git svn rebase" to get new version. On 9 December 2011 05:42, K. John Wu <[email protected]> wrote: > Hi, Petr, > > Thanks for the test data and the instructions to reproduce the > problems. I was able to identify two separate problems in the new > functions that perform aggregation-by-parts. A update set of files > have been checked into the SVN repository at codeforge.lbl.gov. > > Please let me know if you find any additional problems. > > John > > PS: Just in case you need the information, you can check out the > source code with the following command > > svn checkout https://codeforge.lbl.gov/anonscm/fastbit > > Once you have a version checked out, you can simply do 'svn update' to > get the latest version in the repository. > > > On 12/8/11 8:26 AM, Thorgrin wrote: > > Hi John, > > > > thanks for the update, the new features looks great. Unfortunately > > I've bumped into some troubles with the new version. > > > > I've downloaded and compiled the latest version from svn (revision > > 441), I hope that is the latest one. There were no problem compiling > > on my machine, so the previous missing declarations of some functions > > in mine compiler are now OK. > > > > My own app was ending in segfault on aggregation queries, so I tested > > with the ibis tool. Need to say that I'm running the binary ibis tool, > > not the script in examples directory, copied from the machine that the > > library was build on. > > The data that can be used to reproduce the error are here: > > > > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz > > > > the command is: > > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT > > count(*),e0id4,sum(e0id2) WHERE 1=1" > > > > You will notice that this does not indicate any error but does not > > produce any table line either. Previous version always prints at least > > one line (more on verbose). My app ends in segfault after it finishes > > the query and tries to read from the result table, so I believe it is > > the same problem and ibis only reads differently. > > > > In attachement are two logs using -v 10, one for version 1.2.4 (which > > is really svn build 436 or so), one for 1.2.5. Both has extension .out. > > > > When I modify the query only a bit, i get segfault from the ibis tool. > > The command is: > > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q "SELECT > > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" > > > > The valgrind output can be found in file ibis1.2.5.segfault. > > > > It is entirely possible that something is amis on my part, if you have > > any idea what might that be, please let me know. Also if you need any > > other output or information, just let me know. > > > > Petr Velan > > > > > > On 8 December 2011 00:19, K. John Wu <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi, Everyone, > > > > Thanks for all the input on the memory usage issues. There is a > > updated version of FastBit source code marked as ibis1.2.5 with some > > initial implementation of aggregation operations by parts. This > > should reduce the memory usage for aggregation operations that does > > not require all the data to be in memory. > > > > The new version is available at > > > http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 > > < > http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329> > > > > We have also included two patches from Jon Strabala > > and Robert Wong to address a couple of different > > issues. > > > > This release also fixes a number issues raised by > > Alexandre Maure, Anderson Carniel, Jon Strabal, > > Andrew Olson, Teryl Taylor, Robert Wong, and > > others. > > > > Please feel free to let us know if you encounter any problems with > the > > new source code. > > > > John > > > > PS: Just in case you are wondering about which aggregations can be > > performed by parts. The aggregation operators, count, sum, min, and > > max can be performed by parts. The operators, avg, var, and stddev > > can be transformed into sums and counts and are currently internally > > transformed into sums and counts. This transformation may introduce > > very slightly different results. The computations are done with > > double-precision floating-point numbers, which have 16-digit > > precision. The difference should be limited to the last couple of > > digits and should not be noticeable for most users. However, those > > users with very sensitive computations should know that the converted > > expression have different round-off properties from the original > > expression. The bad operations are distinct and median, both which > > require all data to be in memory. > > > > PPS: We have made some progress in supporting joins through a new > > class called quaere. We intend to eventually rename it to > > ibis::query. > > _______________________________________________ > > FastBit-users mailing list > > [email protected] <mailto:[email protected]> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
