Hi Petr,
I have observed similar problems and already identified the bugs and sent fixes to John - the fixes are available in svn 446. Please take a look at the updated sources and let us know if it helps you.
Regards,
Tom
-----Original Message-----
Sent: 16. prosince 2011 11:31
Hi John,
Thanks for your hard work!
I'm sorry I did not notice it earlier, but the aggregation works correctly only when aggregation one column. I'm sure it's just some minor glitch, but it would be great to have it working.
The problem can be reproduced using ibis utility as follows:
This correctly returns one line, first column is 17 just as we asked for.
ibis -d data/fi_hafo1/00000000000[1,5] -q "SELECT e0id4,count(*) WHERE
e0id4 = 17 and e0id11 = 80"
This returns 4 rows, the output seems quite random:
ibis -d data/fi_hafo1/00000000000[1,5] -q "SELECT e0id4,count(*),e0id8 WHERE e0id4 = 17 and e0id11 = 80" -v 2
Also both commands should return same rows in total (11), but that is not so, the second one has two rows missing.
The test data are available at
http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz
I've also noticed that valgrind gives me one invalid read of size 8 each time, but I do not believe this is connected to current issue as it did that in previous version too (ibis.1.2.4.13).
Best regards,
Petr
On 13 December 2011 07:06, K. John Wu <[email protected]> wrote:
> Hi, Petr,
>
> Just made the current code into release 1.2.6
> <https://codeforge.lbl.gov/frs/download.php/370/fastbit-ibis1.2.6.tar.gz>.
> Please give it a try when you get the chance.
>
> In SVN, the tar ball is version 445.
>
> Thanks for all the help.
>
> John
>
>
> On 12/12/11 2:07 AM, Thorgrin wrote:
>> Ho John,
>>
>> Thank you for the bugfix, everything works well now. I would
>> recommend making revision 443 a stable one, instead of current 1.2.5 version.
>>
>> Petr
>>
>> On 9 December 2011 20:51, K. John Wu <[email protected]> wrote:
>>> Hi, Petr,
>>>
>>> Thanks for the feedback. I believe that I have located the source
>>> of the problem -- there are a number of cases I simply neglected to
>>> update the number of rows in the result table ;-)
>>>
>>> Please give SVN version 443 a try when you get the chance. Let me
>>> know if you find anything that still needs attention.
>>>
>>> John
>>>
>>>
>>> On 12/9/11 2:10 AM, Thorgrin wrote:
>>>> Hi, John
>>>>
>>>> Thank you for the quick response. The problems I reported are now
>>>> gone and I'm able to aggregate much larger data thanks to the new feature!
>>>>
>>>> There is one other thing, I've noticed. When aggregating, the
>>>> number of different rows is determined by the first part, so if the
>>>> first part will return only two rows, and second would give me 4,
>>>> when used together I get only the two.
>>>>
>>>> Here are the commands to reproduce:
>>>>
>>>> This gives two rows:
>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5] -q "SELECT
>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1"
>>>>
>>>> This gives four rows:
>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[6] -q "SELECT
>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1"
>>>>
>>>> This works on both parts and gives only two rows.
>>>> ibis -d ~/Downloads/bin/fi_hafo/00000000010[5,6] -q "SELECT
>>>> count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1"
>>>>
>>>> Please find the test data at
>>>> http://www.liberouter.org/~thorgrin/fi_hafo.tar.gz
>>>>
>>>> Petr
>>>>
>>>> PS.: I track the svn in git and I have made some changes so that I
>>>> can easily create an rpm package from the library. Now I only have
>>>> to run "git svn rebase" to get new version.
>>>>
>>>> On 9 December 2011 05:42, K. John Wu <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Hi, Petr,
>>>>
>>>> Thanks for the test data and the instructions to reproduce the
>>>> problems. I was able to identify two separate problems in the
>>>> new
>>>> functions that perform aggregation-by-parts. A update set of
>>>> files
>>>> have been checked into the SVN repository at codeforge.lbl.gov
>>>> <http://codeforge.lbl.gov>.
>>>>
>>>> Please let me know if you find any additional problems.
>>>>
>>>> John
>>>>
>>>> PS: Just in case you need the information, you can check out
>>>> the
>>>> source code with the following command
>>>>
>>>> svn checkout https://codeforge.lbl.gov/anonscm/fastbit
>>>>
>>>> Once you have a version checked out, you can simply do 'svn
>>>> update' to
>>>> get the latest version in the repository.
>>>>
>>>>
>>>> On 12/8/11 8:26 AM, Thorgrin wrote:
>>>> > Hi John,
>>>> >
>>>> > thanks for the update, the new features looks great.
>>>> Unfortunately
>>>> > I've bumped into some troubles with the new version.
>>>> >
>>>> > I've downloaded and compiled the latest version from svn
>>>> (revision
>>>> > 441), I hope that is the latest one. There were no problem
>>>> compiling
>>>> > on my machine, so the previous missing declarations of some
>>>> functions
>>>> > in mine compiler are now OK.
>>>> >
>>>> > My own app was ending in segfault on aggregation queries, so
>>>> I
>>>> tested
>>>> > with the ibis tool. Need to say that I'm running the binary
>>>> ibis
>>>> tool,
>>>> > not the script in examples directory, copied from the machine
>>>> that the
>>>> > library was build on.
>>>> > The data that can be used to reproduce the error are here:
>>>> >
>>>> > http://www.liberouter.org/~thorgrin/fi_hafo1.tar.gz
>>>> <http://www.liberouter.org/%7Ethorgrin/fi_hafo1.tar.gz>
>>>> >
>>>> > the command is:
>>>> > ibis -d ./fi_hafo1/00000000000[1,5] -q "SELECT
>>>> > count(*),e0id4,sum(e0id2) WHERE 1=1"
>>>> >
>>>> > You will notice that this does not indicate any error but
>>>> does not
>>>> > produce any table line either. Previous version always prints
>>>> at
>>>> least
>>>> > one line (more on verbose). My app ends in segfault after it
>>>> finishes
>>>> > the query and tries to read from the result table, so I
>>>> believe
>>>> it is
>>>> > the same problem and ibis only reads differently.
>>>> >
>>>> > In attachement are two logs using -v 10, one for version
>>>> 1.2.4
>>>> (which
>>>> > is really svn build 436 or so), one for 1.2.5. Both has
>>>> extension .out.
>>>> >
>>>> > When I modify the query only a bit, i get segfault from the
>>>> ibis
>>>> tool.
>>>> > The command is:
>>>> > ibis -d ~/Documents/devel/data/fi_hafo1/00000000000[1,5] -q
>>>> "SELECT
>>>> > count(*),e0id4,sum(e0id2),sum(e0id1) WHERE 1=1"
>>>> >
>>>> > The valgrind output can be found in file ibis1.2.5.segfault.
>>>> >
>>>> > It is entirely possible that something is amis on my part, if
>>>> you have
>>>> > any idea what might that be, please let me know. Also if you
>>>> need any
>>>> > other output or information, just let me know.
>>>> >
>>>> > Petr Velan
>>>> >
>>>> >
>>>> > On 8 December 2011 00:19, K. John Wu <[email protected]
>>>> <mailto:[email protected]>
>>>> > <mailto:[email protected] <mailto:[email protected]>>> wrote:
>>>> >
>>>> > Hi, Everyone,
>>>> >
>>>> > Thanks for all the input on the memory usage issues.
>>>> There is a
>>>> > updated version of FastBit source code marked as
>>>> ibis1.2.5
>>>> with some
>>>> > initial implementation of aggregation operations by
>>>> parts. This
>>>> > should reduce the memory usage for aggregation operations
>>>> that does
>>>> > not require all the data to be in memory.
>>>> >
>>>> > The new version is available at
>>>> >
>>>>
>>>>
http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release_
>>>> id=329
>>>>
>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release
>>>> _id=329>
>>>> >
>>>>
>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release
>>>> _id=329
>>>>
>>>> <http://codeforge.lbl.gov/project/showfiles.php?group_id=44&release
>>>> _id=329>>
>>>> >
>>>> > We have also included two patches from Jon Strabala
>>>> > and Robert Wong to address a couple of different
>>>> > issues.
>>>> >
>>>> > This release also fixes a number issues raised by
>>>> > Alexandre Maure, Anderson Carniel, Jon Strabal,
>>>> > Andrew Olson, Teryl Taylor, Robert Wong, and
>>>> > others.
>>>> >
>>>> > Please feel free to let us know if you encounter any
>>>> problems with the
>>>> > new source code.
>>>> >
>>>> > John
>>>> >
>>>> > PS: Just in case you are wondering about which
>>>> aggregations
>>>> can be
>>>> > performed by parts. The aggregation operators, count,
>>>> sum,
>>>> min, and
>>>> > max can be performed by parts. The operators, avg, var,
>>>> and
>>>> stddev
>>>> > can be transformed into sums and counts and are currently
>>>> internally
>>>> > transformed into sums and counts. This transformation
>>>> may
>>>> introduce
>>>> > very slightly different results. The computations are
>>>> done with
>>>> > double-precision floating-point numbers, which have
>>>> 16-digit
>>>> > precision. The difference should be limited to the last
>>>> couple of
>>>> > digits and should not be noticeable for most users.
>>>> However, those
>>>> > users with very sensitive computations should know that
>>>> the
>>>> converted
>>>> > _expression_ have different round-off properties from the
>>>> original
>>>> > _expression_. The bad operations are distinct and median,
>>>> both which
>>>> > require all data to be in memory.
>>>> >
>>>> > PPS: We have made some progress in supporting joins
>>>> through
>>>> a new
>>>> > class called quaere. We intend to eventually rename it
>>>> to
>>>> > ibis::query.
>>>> > _______________________________________________
>>>> > FastBit-users mailing list
>>>> > [email protected]
>>>> <mailto:[email protected]>
>>>> <mailto:[email protected]
>>>> <mailto:[email protected]>>
>>>> >
>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > FastBit-users mailing list
>>>> > [email protected]
>>>> <mailto:[email protected]>
>>>> >
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>>>
>>>>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
