Hi, Petr, Thanks for the suggestions. I have looked into the output from valgrind on a Mac. It looks to be that all the complaints are regarding some system library functions such as uninitialized reads are in getpwuid_r and pthread_rwlock_init, and memory leaks are related to vnprintf. These could be because these system function can not be tracked by valgrind correctly. If anyone knows what might actually be going, please let me know.
In the mean time, I will try valgrind again on a different type of machine tomorrow and see if I can get a different set of print out.. John On 12/20/11 2:34 PM, Thorgrin wrote: > Hi John, > > I've tested the aggregation some more and it works really well. There > are no performance problems with the new version that I have noticed, > it might have been someone else. (Or I just don't remember, in which > case please point me to the right message ;-)). > > There are few minor things, not really important but might be worth a > notice, before releasing next stable version. I use valgrind to check > for memory leaks and other memory-related problems and I noticed that > there are always some uses of uninitialised values and invalid reads. > I do not have the library with debug symbols right now, so the output > would is quite limited, but it might give the hint. If you are not > getting similar output, the fault might be mine. Please find the > valgrind output in valgrind.txt attachment. > > The other issue is that when I put in an query condition on column, > that does not exist in the table, the table that is filtered out leaks > memory. I just discovered this and it might be some fault in my own > code, but it should be easy to verify. It seems to me that the > table::select function returns NULL in such cases without freeing up > the memory. It would be great if someone could confirm or disprove > this. The valgrind output (--leak-check=full) is in leak.txt > attachment. > > That is all from me now, thank you for all the work you've put into > this great library. > > Petr > > On 20 December 2011 22:30, K. John Wu <[email protected]> wrote: >> Hi, Petr, >> >> Hope things are moving along smoothly now. Were you the one who >> brought some problem with performance issues with the new version? If >> yes, do you have any more specific information for me? I am trying to >> wrap the current round of changes and would like to get this issue >> resolved if it is still bothering you. >> >> Thanks. >> >> John >> >> >> On 12/19/11 4:20 AM, Thorgrin wrote: >>> Hi Tomas, >>> >>> thank you for your bugfix, it works like a charm. My bad that I missed >>> the new svn revision before asking. >>> >>> Petr >>> >>> On 16 December 2011 16:26, Rybka, Tomas <[email protected]> wrote: >>>> Hi Petr, >>>> >>>> I have observed similar problems and already identified the bugs and sent >>>> fixes to John - the fixes are available in svn 446. Please take a look at >>>> the updated sources and let us know if it helps you. >>>> >>>> Regards, >>>> Tom >>>> >>>> >>>> >>>> >>>> -----Original Message----- >>>> Sent: 16. prosince 2011 11:31 >>>> >>>> Hi John, >>>> >>>> Thanks for your hard work! >>>> >>>> I'm sorry I did not notice it earlier, but the aggregation works correctly >>>> only when aggregation one column. I'm sure it's just some minor glitch, but >>>> it would be great to have it working. >>>> >>>> The problem can be reproduced using ibis utility as follows: >>>> >>>> This correctly returns one line, first column is 17 just as we asked for. >>>> ibis -d data/fi_hafo1/00000000000[1,5] -q "SELECT e0id4,count(*) WHERE >>>> e0id4 = 17 and e0id11 = 80" >>>> >>>> This returns 4 rows, the output seems quite random: >>>> ibis -d data/fi_hafo1/00000000000[1,5] -q "SELECT e0id4,count(*),e0id8 >>>> WHERE >>>> e0id4 = 17 and e0id11 = 80" -v 2 >>>> >>>> Also both commands should return same rows in total (11), but that is not >>>> so, the second one has two rows missing. >>>> >>>> The test data are available at >>>> http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz >>>> >>>> I've also noticed that valgrind gives me one invalid read of size 8 each >>>> time, but I do not believe this is connected to current issue as it did >>>> that >>>> in previous version too (ibis.1.2.4.13). >>>> >>>> Best regards, >>>> Petr >>>> >>>> On 13 December 2011 07:06, K. John Wu <[email protected]> wrote: >>>>> Hi, Petr, >>>>> >>>>> Just made the current code into release 1.2.6 >>>>> <https://codeforge.lbl.gov/frs/download.php/370/fastbit-ibis1.2.6.tar.gz>. >>>>> Please give it a try when you get the chance. >>>>> >>>>> In SVN, the tar ball is version 445. >>>>> >>>>> Thanks for all the help. >>>>> >>>>> John >>>>> >>>>> >>>>> On 12/12/11 2:07 AM, Thorgrin wrote: >>>>>> Ho John, >>>>>> >>>>>> Thank you for the bugfix, everything works well now. I would >>>>>> recommend making revision 443 a stable one, instead of current 1.2.5 >>>>>> version. >>>>>> >>>>>> Petr >>>>>> >>>>>> On 9 December 2011 20:51, K. John Wu <[email protected]> wrote: >>>>>>> Hi, Petr, >>>>>>> >>>>>>> Thanks for the feedback. I believe that I have located the source >>>>>>> of the problem -- there are a number of cases I simply neglected to >>>>>>> update the number of rows in the result table ;-) >>>>>>> >>>>>>> Please give SVN version 443 a try when you get the chance. Let me >>>>>>> know if you find anything that still needs attention. >>>>>>> >>>>>>> John >>>>>>> >>>>>>> >>>>>>> On 12/9/11 2:10 AM, Thorgrin wrote: >>>>>>>> Hi, John >>>>>>>> >>>>>>>> Thank you for the quick response. The problems I reported are now >>>>>>>> gone and I'm able to aggregate much larger data thanks to the new >>>>>>>> feature! >>>>>>>> >>>>>>>> There is one other thing, I've noticed. When aggregating, the >>>>>>>> number of different rows is determined by the first part, so if the >>>>>>>> first part will return only two rows, and second would give me 4, >>>>>>>> when used together I get only the two. >>>>>>>> >>>>>>>> Here are the commands to reproduce: >>>>>>>> >>>>>>>> This gives two rows: >>>>>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT >>>>>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>>>>>>> >>>>>>>> This gives four rows: >>>>>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT >>>>>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>>>>>>> >>>>>>>> This works on both parts and gives only two rows. >>>>>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT >>>>>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>>>>>>> >>>>>>>> Please find the test data at >>>>>>>> http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz >>>>>>>> >>>>>>>> Petr >>>>>>>> >>>>>>>> PS.: I track the svn in git and I have made some changes so that I >>>>>>>> can easily create an rpm package from the library. Now I only have >>>>>>>> to run "git svn rebase" to get new version. >>>>>>>> >>>>>>>> On 9 December 2011 05:42, K. John Wu <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi, Petr, >>>>>>>> >>>>>>>> Thanks for the test data and the instructions to reproduce the >>>>>>>> problems. I was able to identify two separate problems in the >>>>>>>> new >>>>>>>> functions that perform aggregation-by-parts. A update set of >>>>>>>> files >>>>>>>> have been checked into the SVN repository at codeforge.lbl.gov >>>>>>>> <http://codeforge.lbl.gov>. >>>>>>>> >>>>>>>> Please let me know if you find any additional problems. >>>>>>>> >>>>>>>> John >>>>>>>> >>>>>>>> PS: Just in case you need the information, you can check out >>>>>>>> the >>>>>>>> source code with the following command >>>>>>>> >>>>>>>> svn checkout https://codeforge.lbl.gov/anonscm/fastbit >>>>>>>> >>>>>>>> Once you have a version checked out, you can simply do 'svn >>>>>>>> update' to >>>>>>>> get the latest version in the repository. >>>>>>>> >>>>>>>> >>>>>>>> On 12/8/11 8:26 AM, Thorgrin wrote: >>>>>>>> > Hi John, >>>>>>>> > >>>>>>>> > thanks for the update, the new features looks great. >>>>>>>> Unfortunately >>>>>>>> > I've bumped into some troubles with the new version. >>>>>>>> > >>>>>>>> > I've downloaded and compiled the latest version from svn >>>>>>>> (revision >>>>>>>> > 441), I hope that is the latest one. There were no problem >>>>>>>> compiling >>>>>>>> > on my machine, so the previous missing declarations of some >>>>>>>> functions >>>>>>>> > in mine compiler are now OK. >>>>>>>> > >>>>>>>> > My own app was ending in segfault on aggregation queries, so >>>>>>>> I >>>>>>>> tested >>>>>>>> > with the ibis tool. Need to say that I'm running the binary >>>>>>>> ibis >>>>>>>> tool, >>>>>>>> > not the script in examples directory, copied from the machine >>>>>>>> that the >>>>>>>> > library was build on. >>>>>>>> > The data that can be used to reproduce the error are here: >>>>>>>> > >>>>>>>> > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz >>>>>>>> <http://www.liberouter.org/%7Ethorgrin/fi_hafo1.tar.gz> >>>>>>>> > >>>>>>>> > the command is: >>>>>>>> > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT >>>>>>>> > count(*),e0id4,sum(e0id2) WHERE 1=1" >>>>>>>> > >>>>>>>> > You will notice that this does not indicate any error but >>>>>>>> does not >>>>>>>> > produce any table line either. Previous version always prints >>>>>>>> at >>>>>>>> least >>>>>>>> > one line (more on verbose). My app ends in segfault after it >>>>>>>> finishes >>>>>>>> > the query and tries to read from the result table, so I >>>>>>>> believe >>>>>>>> it is >>>>>>>> > the same problem and ibis only reads differently. >>>>>>>> > >>>>>>>> > In attachement are two logs using -v 10, one for version >>>>>>>> 1.2.4 >>>>>>>> (which >>>>>>>> > is really svn build 436 or so), one for 1.2.5. Both has >>>>>>>> extension .out. >>>>>>>> > >>>>>>>> > When I modify the query only a bit, i get segfault from the >>>>>>>> ibis >>>>>>>> tool. >>>>>>>> > The command is: >>>>>>>> > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q >>>>>>>> "SELECT >>>>>>>> > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1" >>>>>>>> > >>>>>>>> > The valgrind output can be found in file ibis1.2.5.segfault. >>>>>>>> > >>>>>>>> > It is entirely possible that something is amis on my part, if >>>>>>>> you have >>>>>>>> > any idea what might that be, please let me know. Also if you >>>>>>>> need any >>>>>>>> > other output or information, just let me know. >>>>>>>> > >>>>>>>> > Petr Velan >>>>>>>> > >>>>>>>> > >>>>>>>> > On 8 December 2011 00:19, K. John Wu <[email protected] >>>>>>>> <mailto:[email protected]> >>>>>>>> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >>>>>>>> > >>>>>>>> > Hi, Everyone, >>>>>>>> > >>>>>>>> > Thanks for all the input on the memory usage issues. >>>>>>>> There is a >>>>>>>> > updated version of FastBit source code marked as >>>>>>>> ibis1.2.5 >>>>>>>> with some >>>>>>>> > initial implementation of aggregation operations by >>>>>>>> parts. This >>>>>>>> > should reduce the memory usage for aggregation operations >>>>>>>> that does >>>>>>>> > not require all the data to be in memory. >>>>>>>> > >>>>>>>> > The new version is available at >>>>>>>> > >>>>>>>> >>>>>>>> http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_ >>>>>>>> id=329 >>>>>>>> >>>>>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release >>>>>>>> _id=329> >>>>>>>> > >>>>>>>> >>>>>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release >>>>>>>> _id=329 >>>>>>>> >>>>>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release >>>>>>>> _id=329>> >>>>>>>> > >>>>>>>> > We have also included two patches from Jon Strabala >>>>>>>> > and Robert Wong to address a couple of different >>>>>>>> > issues. >>>>>>>> > >>>>>>>> > This release also fixes a number issues raised by >>>>>>>> > Alexandre Maure, Anderson Carniel, Jon Strabal, >>>>>>>> > Andrew Olson, Teryl Taylor, Robert Wong, and >>>>>>>> > others. >>>>>>>> > >>>>>>>> > Please feel free to let us know if you encounter any >>>>>>>> problems with the >>>>>>>> > new source code. >>>>>>>> > >>>>>>>> > John >>>>>>>> > >>>>>>>> > PS: Just in case you are wondering about which >>>>>>>> aggregations >>>>>>>> can be >>>>>>>> > performed by parts. The aggregation operators, count, >>>>>>>> sum, >>>>>>>> min, and >>>>>>>> > max can be performed by parts. The operators, avg, var, >>>>>>>> and >>>>>>>> stddev >>>>>>>> > can be transformed into sums and counts and are currently >>>>>>>> internally >>>>>>>> > transformed into sums and counts. This transformation >>>>>>>> may >>>>>>>> introduce >>>>>>>> > very slightly different results. The computations are >>>>>>>> done with >>>>>>>> > double-precision floating-point numbers, which have >>>>>>>> 16-digit >>>>>>>> > precision. The difference should be limited to the last >>>>>>>> couple of >>>>>>>> > digits and should not be noticeable for most users. >>>>>>>> However, those >>>>>>>> > users with very sensitive computations should know that >>>>>>>> the >>>>>>>> converted >>>>>>>> > expression have different round-off properties from the >>>>>>>> original >>>>>>>> > expression. The bad operations are distinct and median, >>>>>>>> both which >>>>>>>> > require all data to be in memory. >>>>>>>> > >>>>>>>> > PPS: We have made some progress in supporting joins >>>>>>>> through >>>>>>>> a new >>>>>>>> > class called quaere. We intend to eventually rename it >>>>>>>> to >>>>>>>> > ibis::query. >>>>>>>> > _______________________________________________ >>>>>>>> > FastBit-users mailing list >>>>>>>> > [email protected] >>>>>>>> <mailto:[email protected]> >>>>>>>> <mailto:[email protected] >>>>>>>> <mailto:[email protected]>> >>>>>>>> > >>>>>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > FastBit-users mailing list >>>>>>>> > [email protected] >>>>>>>> <mailto:[email protected]> >>>>>>>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>>>>>> >>>>>>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
