Ho John, Thank you for the bugfix, everything works well now. I would recommend making revision 443 a stable one, instead of current 1.2.5 version.
Petr On 9 December 2011 20:51, K. John Wu <[email protected]> wrote: > Hi, Petr, > > Thanks for the feedback. I believe that I have located the source of > the problem -- there are a number of cases I simply neglected to > update the number of rows in the result table ;-) > > Please give SVN version 443 a try when you get the chance. Let me > know if you find anything that still needs attention. > > John > > > On 12/9/11 2:10 AM, Thorgrin wrote: >> Hi, John >> >> Thank you for the quick response. The problems I reported are now gone >> and I'm able to aggregate much larger data thanks to the new feature! >> >> There is one other thing, I've noticed. When aggregating, the number >> of different rows is determined by the first part, so if the first >> part will return only two rows, and second would give me 4, when used >> together I get only the two. >> >> Here are the commands to reproduce: >> >> This gives two rows: >> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT >> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >> >> This gives four rows: >> ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT >> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >> >> This works on both parts and gives only two rows. >> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT >> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >> >> Please find the test data at >> http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz >> >> Petr >> >> PS.: I track the svn in git and I have made some changes so that I can >> easily create an rpm package from the library. Now I only have to run >> "git svn rebase" to get new version. >> >> On 9 December 2011 05:42, K. John Wu <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi, Petr, >> >> Thanks for the test data and the instructions to reproduce the >> problems. I was able to identify two separate problems in the new >> functions that perform aggregation-by-parts. A update set of files >> have been checked into the SVN repository at codeforge.lbl.gov >> <http://codeforge.lbl.gov>. >> >> Please let me know if you find any additional problems. >> >> John >> >> PS: Just in case you need the information, you can check out the >> source code with the following command >> >> svn checkout https://codeforge.lbl.gov/anonscm/fastbit >> >> Once you have a version checked out, you can simply do 'svn update' to >> get the latest version in the repository. >> >> >> On 12/8/11 8:26 AM, Thorgrin wrote: >> > Hi John, >> > >> > thanks for the update, the new features looks great. Unfortunately >> > I've bumped into some troubles with the new version. >> > >> > I've downloaded and compiled the latest version from svn (revision >> > 441), I hope that is the latest one. There were no problem compiling >> > on my machine, so the previous missing declarations of some >> functions >> > in mine compiler are now OK. >> > >> > My own app was ending in segfault on aggregation queries, so I >> tested >> > with the ibis tool. Need to say that I'm running the binary ibis >> tool, >> > not the script in examples directory, copied from the machine >> that the >> > library was build on. >> > The data that can be used to reproduce the error are here: >> > >> > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz >> <http://www.liberouter.org/%7Ethorgrin/fi_hafo1.tar.gz> >> > >> > the command is: >> > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT >> > count(*),e0id4,sum(e0id2) WHERE 1=1" >> > >> > You will notice that this does not indicate any error but does not >> > produce any table line either. Previous version always prints at >> least >> > one line (more on verbose). My app ends in segfault after it >> finishes >> > the query and tries to read from the result table, so I believe >> it is >> > the same problem and ibis only reads differently. >> > >> > In attachement are two logs using -v 10, one for version 1.2.4 >> (which >> > is really svn build 436 or so), one for 1.2.5. Both has >> extension .out. >> > >> > When I modify the query only a bit, i get segfault from the ibis >> tool. >> > The command is: >> > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q "SELECT >> > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >> > >> > The valgrind output can be found in file ibis1.2.5.segfault. >> > >> > It is entirely possible that something is amis on my part, if >> you have >> > any idea what might that be, please let me know. Also if you >> need any >> > other output or information, just let me know. >> > >> > Petr Velan >> > >> > >> > On 8 December 2011 00:19, K. John Wu <[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >> > Hi, Everyone, >> > >> > Thanks for all the input on the memory usage issues. There is a >> > updated version of FastBit source code marked as ibis1.2.5 >> with some >> > initial implementation of aggregation operations by parts. This >> > should reduce the memory usage for aggregation operations >> that does >> > not require all the data to be in memory. >> > >> > The new version is available at >> > >> http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 >> >> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329> >> > >> >> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 >> >> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329>> >> > >> > We have also included two patches from Jon Strabala >> > and Robert Wong to address a couple of different >> > issues. >> > >> > This release also fixes a number issues raised by >> > Alexandre Maure, Anderson Carniel, Jon Strabal, >> > Andrew Olson, Teryl Taylor, Robert Wong, and >> > others. >> > >> > Please feel free to let us know if you encounter any >> problems with the >> > new source code. >> > >> > John >> > >> > PS: Just in case you are wondering about which aggregations >> can be >> > performed by parts. The aggregation operators, count, sum, >> min, and >> > max can be performed by parts. The operators, avg, var, and >> stddev >> > can be transformed into sums and counts and are currently >> internally >> > transformed into sums and counts. This transformation may >> introduce >> > very slightly different results. The computations are done with >> > double-precision floating-point numbers, which have 16-digit >> > precision. The difference should be limited to the last >> couple of >> > digits and should not be noticeable for most users. >> However, those >> > users with very sensitive computations should know that the >> converted >> > expression have different round-off properties from the original >> > expression. The bad operations are distinct and median, >> both which >> > require all data to be in memory. >> > >> > PPS: We have made some progress in supporting joins through >> a new >> > class called quaere. We intend to eventually rename it to >> > ibis::query. >> > _______________________________________________ >> > FastBit-users mailing list >> > [email protected] >> <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>> >> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > >> > >> > >> > >> > _______________________________________________ >> > FastBit-users mailing list >> > [email protected] <mailto:[email protected]> >> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> >> _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
