I was able to resolve the issue by setting the stack size using the JVM argument -Xss rather than ulimit or setrlimit, since we are calling fast bit through JNA/JNI.
On 9/5/13 5:56 PM, "Enns, Steven" <[email protected]> wrote: >On the topic of increasing the stack size, it's curious that we are able >to recurse on the same parse tree without stack overflow in java... > >On 9/5/13 5:51 PM, "Enns, Steven" <[email protected]> wrote: > >>Thanks John and Dominique! >> >>Unfortunately our column has a keyword index, so we are unable to use >>"in" >>queries. We will try to balance our parentheses. >> >>Have either of you had success increasing the stack size? I tried using >>ulimit and setrlimit, but neither seemed to work. >> >>John - Have you considered changing the methods that operate on the parse >>tree to be iterative rather than recursive? I started looking at >>ibis::qExpr::simplify and I noticed two challenges: >> >>1. ibis::qExpr::simplify requires post-order traversal, which is somewhat >>unnatural to implement iteratively >>2. There are possibly many other methods which would need to be modified >> >> >>On 9/5/13 5:17 PM, "K. John Wu" <[email protected]> wrote: >> >>>Hi, Dominique and Steve, >>> >>>If your long list of ORs are most related to a single column (or a >>>small number of columns), then it is possible to convert them into an >>>expression of the form >>> >>>column_name IN (list_of_strings_or_numbers ) >>> >>>The list in the parentheses has to be either a list of numbers of a >>>list of strings (a mixture will be treated as a list of strings). >>> >>>Hope this helps. >>> >>>John >>> >>>PS: more information about expressions of a where clause is available at >>> >>>http://lbl.gov/~kwu/fastbit/doc/html/classibis_1_1whereClause.html >>> >>> >>>On 9/5/13 3:39 PM, Prunier, Dominique wrote: >>>> Steve, >>>> >>>> >>>> >>>> We faced the same issue and we worked around that by carefully >>>> balancing the big OR with parenthesis so that you don¹t go deeper that >>>> log2(OR count). >>>> >>>> >>>> >>>> Hope this helps ! >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> *From:*[email protected] >>>> [mailto:[email protected]] *On Behalf Of *Enns, >>>>Steven >>>> *Sent:* Thursday, September 05, 2013 6:16 PM >>>> *To:* FastBit Users >>>> *Subject:* [FastBit-users] stack overflow on large queries >>>> >>>> >>>> >>>> Hi John, >>>> >>>> >>>> >>>> I am running a query of 2140 OR clauses and observing segmentation >>>> fault in ibis::qExpr::simplify(ibis::qExpr*&) at recursive depth of >>>> 1087. Could you help us debug this? >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Steve >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>>_______________________________________________ >>>FastBit-users mailing list >>>[email protected] >>>https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> >>_______________________________________________ >>FastBit-users mailing list >>[email protected] >>https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
