Hi, Petr, Thanks for the feedback. I believe that I have located the source of the problem -- there are a number of cases I simply neglected to update the number of rows in the result table ;-)
Please give SVN version 443 a try when you get the chance. Let me know if you find anything that still needs attention. John On 12/9/11 2:10 AM, Thorgrin wrote: > Hi, John > > Thank you for the quick response. The problems I reported are now gone > and I'm able to aggregate much larger data thanks to the new feature! > > There is one other thing, I've noticed. When aggregating, the number > of different rows is determined by the first part, so if the first > part will return only two rows, and second would give me 4, when used > together I get only the two. > > Here are the commands to reproduce: > > This gives two rows: > ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" > > This gives four rows: > ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" > > This works on both parts and gives only two rows. > ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" > > Please find the test data at > http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz > > Petr > > PS.: I track the svn in git and I have made some changes so that I can > easily create an rpm package from the library. Now I only have to run > "git svn rebase" to get new version. > > On 9 December 2011 05:42, K. John Wu <[email protected] > <mailto:[email protected]>> wrote: > > Hi, Petr, > > Thanks for the test data and the instructions to reproduce the > problems. I was able to identify two separate problems in the new > functions that perform aggregation-by-parts. A update set of files > have been checked into the SVN repository at codeforge.lbl.gov > <http://codeforge.lbl.gov>. > > Please let me know if you find any additional problems. > > John > > PS: Just in case you need the information, you can check out the > source code with the following command > > svn checkout https://codeforge.lbl.gov/anonscm/fastbit > > Once you have a version checked out, you can simply do 'svn update' to > get the latest version in the repository. > > > On 12/8/11 8:26 AM, Thorgrin wrote: > > Hi John, > > > > thanks for the update, the new features looks great. Unfortunately > > I've bumped into some troubles with the new version. > > > > I've downloaded and compiled the latest version from svn (revision > > 441), I hope that is the latest one. There were no problem compiling > > on my machine, so the previous missing declarations of some > functions > > in mine compiler are now OK. > > > > My own app was ending in segfault on aggregation queries, so I > tested > > with the ibis tool. Need to say that I'm running the binary ibis > tool, > > not the script in examples directory, copied from the machine > that the > > library was build on. > > The data that can be used to reproduce the error are here: > > > > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz > <http://www.liberouter.org/%7Ethorgrin/fi_hafo1.tar.gz> > > > > the command is: > > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT > > count(*),e0id4,sum(e0id2) WHERE 1=1" > > > > You will notice that this does not indicate any error but does not > > produce any table line either. Previous version always prints at > least > > one line (more on verbose). My app ends in segfault after it > finishes > > the query and tries to read from the result table, so I believe > it is > > the same problem and ibis only reads differently. > > > > In attachement are two logs using -v 10, one for version 1.2.4 > (which > > is really svn build 436 or so), one for 1.2.5. Both has > extension .out. > > > > When I modify the query only a bit, i get segfault from the ibis > tool. > > The command is: > > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q "SELECT > > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" > > > > The valgrind output can be found in file ibis1.2.5.segfault. > > > > It is entirely possible that something is amis on my part, if > you have > > any idea what might that be, please let me know. Also if you > need any > > other output or information, just let me know. > > > > Petr Velan > > > > > > On 8 December 2011 00:19, K. John Wu <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > Hi, Everyone, > > > > Thanks for all the input on the memory usage issues. There is a > > updated version of FastBit source code marked as ibis1.2.5 > with some > > initial implementation of aggregation operations by parts. This > > should reduce the memory usage for aggregation operations > that does > > not require all the data to be in memory. > > > > The new version is available at > > > http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 > > <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329> > > > <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 > > <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329>> > > > > We have also included two patches from Jon Strabala > > and Robert Wong to address a couple of different > > issues. > > > > This release also fixes a number issues raised by > > Alexandre Maure, Anderson Carniel, Jon Strabal, > > Andrew Olson, Teryl Taylor, Robert Wong, and > > others. > > > > Please feel free to let us know if you encounter any > problems with the > > new source code. > > > > John > > > > PS: Just in case you are wondering about which aggregations > can be > > performed by parts. The aggregation operators, count, sum, > min, and > > max can be performed by parts. The operators, avg, var, and > stddev > > can be transformed into sums and counts and are currently > internally > > transformed into sums and counts. This transformation may > introduce > > very slightly different results. The computations are done with > > double-precision floating-point numbers, which have 16-digit > > precision. The difference should be limited to the last > couple of > > digits and should not be noticeable for most users. > However, those > > users with very sensitive computations should know that the > converted > > expression have different round-off properties from the original > > expression. The bad operations are distinct and median, > both which > > require all data to be in memory. > > > > PPS: We have made some progress in supporting joins through > a new > > class called quaere. We intend to eventually rename it to > > ibis::query. > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > > > > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] <mailto:[email protected]> > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
