Hi, Petr, Just made the current code into release 1.2.6 <https://codeforge.lbl.gov/frs/download.php/370/fastbit-ibis1.2.6.tar.gz>. Please give it a try when you get the chance.
In SVN, the tar ball is version 445. Thanks for all the help. John On 12/12/11 2:07 AM, Thorgrin wrote: > Ho John, > > Thank you for the bugfix, everything works well now. I would recommend > making revision 443 a stable one, instead of current 1.2.5 version. > > Petr > > On 9 December 2011 20:51, K. John Wu <[email protected]> wrote: >> Hi, Petr, >> >> Thanks for the feedback. I believe that I have located the source of >> the problem -- there are a number of cases I simply neglected to >> update the number of rows in the result table ;-) >> >> Please give SVN version 443 a try when you get the chance. Let me >> know if you find anything that still needs attention. >> >> John >> >> >> On 12/9/11 2:10 AM, Thorgrin wrote: >>> Hi, John >>> >>> Thank you for the quick response. The problems I reported are now gone >>> and I'm able to aggregate much larger data thanks to the new feature! >>> >>> There is one other thing, I've noticed. When aggregating, the number >>> of different rows is determined by the first part, so if the first >>> part will return only two rows, and second would give me 4, when used >>> together I get only the two. >>> >>> Here are the commands to reproduce: >>> >>> This gives two rows: >>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT >>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>> >>> This gives four rows: >>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT >>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>> >>> This works on both parts and gives only two rows. >>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT >>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>> >>> Please find the test data at >>> http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz >>> >>> Petr >>> >>> PS.: I track the svn in git and I have made some changes so that I can >>> easily create an rpm package from the library. Now I only have to run >>> "git svn rebase" to get new version. >>> >>> On 9 December 2011 05:42, K. John Wu <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi, Petr, >>> >>> Thanks for the test data and the instructions to reproduce the >>> problems. I was able to identify two separate problems in the new >>> functions that perform aggregation-by-parts. A update set of files >>> have been checked into the SVN repository at codeforge.lbl.gov >>> <http://codeforge.lbl.gov>. >>> >>> Please let me know if you find any additional problems. >>> >>> John >>> >>> PS: Just in case you need the information, you can check out the >>> source code with the following command >>> >>> svn checkout https://codeforge.lbl.gov/anonscm/fastbit >>> >>> Once you have a version checked out, you can simply do 'svn update' to >>> get the latest version in the repository. >>> >>> >>> On 12/8/11 8:26 AM, Thorgrin wrote: >>> > Hi John, >>> > >>> > thanks for the update, the new features looks great. Unfortunately >>> > I've bumped into some troubles with the new version. >>> > >>> > I've downloaded and compiled the latest version from svn (revision >>> > 441), I hope that is the latest one. There were no problem compiling >>> > on my machine, so the previous missing declarations of some >>> functions >>> > in mine compiler are now OK. >>> > >>> > My own app was ending in segfault on aggregation queries, so I >>> tested >>> > with the ibis tool. Need to say that I'm running the binary ibis >>> tool, >>> > not the script in examples directory, copied from the machine >>> that the >>> > library was build on. >>> > The data that can be used to reproduce the error are here: >>> > >>> > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz >>> <http://www.liberouter.org/%7Ethorgrin/fi_hafo1.tar.gz> >>> > >>> > the command is: >>> > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT >>> > count(*),e0id4,sum(e0id2) WHERE 1=1" >>> > >>> > You will notice that this does not indicate any error but does not >>> > produce any table line either. Previous version always prints at >>> least >>> > one line (more on verbose). My app ends in segfault after it >>> finishes >>> > the query and tries to read from the result table, so I believe >>> it is >>> > the same problem and ibis only reads differently. >>> > >>> > In attachement are two logs using -v 10, one for version 1.2.4 >>> (which >>> > is really svn build 436 or so), one for 1.2.5. Both has >>> extension .out. >>> > >>> > When I modify the query only a bit, i get segfault from the ibis >>> tool. >>> > The command is: >>> > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q "SELECT >>> > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>> > >>> > The valgrind output can be found in file ibis1.2.5.segfault. >>> > >>> > It is entirely possible that something is amis on my part, if >>> you have >>> > any idea what might that be, please let me know. Also if you >>> need any >>> > other output or information, just let me know. >>> > >>> > Petr Velan >>> > >>> > >>> > On 8 December 2011 00:19, K. John Wu <[email protected] >>> <mailto:[email protected]> >>> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >>> > >>> > Hi, Everyone, >>> > >>> > Thanks for all the input on the memory usage issues. There is a >>> > updated version of FastBit source code marked as ibis1.2.5 >>> with some >>> > initial implementation of aggregation operations by parts. This >>> > should reduce the memory usage for aggregation operations >>> that does >>> > not require all the data to be in memory. >>> > >>> > The new version is available at >>> > >>> >>> http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 >>> >>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329> >>> > >>> >>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329 >>> >>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_id=329>> >>> > >>> > We have also included two patches from Jon Strabala >>> > and Robert Wong to address a couple of different >>> > issues. >>> > >>> > This release also fixes a number issues raised by >>> > Alexandre Maure, Anderson Carniel, Jon Strabal, >>> > Andrew Olson, Teryl Taylor, Robert Wong, and >>> > others. >>> > >>> > Please feel free to let us know if you encounter any >>> problems with the >>> > new source code. >>> > >>> > John >>> > >>> > PS: Just in case you are wondering about which aggregations >>> can be >>> > performed by parts. The aggregation operators, count, sum, >>> min, and >>> > max can be performed by parts. The operators, avg, var, and >>> stddev >>> > can be transformed into sums and counts and are currently >>> internally >>> > transformed into sums and counts. This transformation may >>> introduce >>> > very slightly different results. The computations are done with >>> > double-precision floating-point numbers, which have 16-digit >>> > precision. The difference should be limited to the last >>> couple of >>> > digits and should not be noticeable for most users. >>> However, those >>> > users with very sensitive computations should know that the >>> converted >>> > expression have different round-off properties from the original >>> > expression. The bad operations are distinct and median, >>> both which >>> > require all data to be in memory. >>> > >>> > PPS: We have made some progress in supporting joins through >>> a new >>> > class called quaere. We intend to eventually rename it to >>> > ibis::query. >>> > _______________________________________________ >>> > FastBit-users mailing list >>> > [email protected] >>> <mailto:[email protected]> >>> <mailto:[email protected] >>> <mailto:[email protected]>> >>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> > >>> > >>> > >>> > >>> > _______________________________________________ >>> > FastBit-users mailing list >>> > [email protected] <mailto:[email protected]> >>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >>> _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
