Hi, Alan, FastBit is employing a poor algorithm to compute the sums right now, and it requires a lot of memory. In your particular case, you might want to break the query into a number of smaller pieces, for example, by IPV4_SRC_ADDR ranges with something like
ibis -d . -q "select sum(in_bytes), ipv4_src_addr, ipv4_dst_addr where in_bytes > 0 && ipv4_src_addr < 1e8" ibis -d . -q "select sum(in_bytes), ipv4_src_addr, ipv4_dst_addr where in_bytes > 0 && 1e8 <= ipv4_src_addr < 2e8" Note that you probably want to bin the IP addresses somehow. John On 8/11/11 3:16 AM, Alan Kemp wrote: > Hi All > > I seem to get a memory error when running ibis on a 54G 53 million row > fastbit db. > > # ibis -d . -q "SELECT sum(IN_BYTES),IPV4_SRC_ADDR,IPV4_DST_ADDR where > IN_BYTES > 0" > > Warning -- fileManager::unload timed-out while waiting for 281,791,252 > bytes (totalBytes=1,799,361,864, maxBytes=2,076,628,992) Error -- > bundles::ctor received an exception, start cleaning up Warning -- > table::select absorbed a bad_alloc exception (storage::ctor(copy > memory) failed), will return a nil pointer > > Is there something on the IS I can tune or to be able to query larger > datasets. I'm running Linux ubuntu (patty) > > I am writing about 480 000 records every 5 minutes into 5 minute > partitions. > > This is the header of that -part.txt file on how I create the records. > > --snip-- > BEGIN HEADER > Name = "00" > Description = "/usr/local/bin//ardea -d > /usr/local/iris/data/flows/flowdb/2011/08/11/01/00 -m > FIRST_SWITCHED:uint LAST_SWITCHED:uint IPV4_SRC_ADDR:uint > IPV4_DST_ADDR:uint IPV4_SRC_PORT:uint IPV4_DST_PORT:uint PROTOCOL:key > SRC_TOS:uint IN_PKTS:uint IN_BYTES:uint OUT_PKTS:uint OUT_BYTES:uint > INPUT_SNMP:key OUTPUT_SNMP:key SRC_AS:uint DST_AS:uint > IPV4_SRC_MASK:uint IPV4_DST_MASK:uint DST_TOS:uint IPV4_NEXTHOP:uint > EXADDR:uint AS_PATH_1:uint AS_PATH_2:uint AS_PATH_3:uint > AS_PATH_4:uint AS_PATH_5:uint -t /dev/stdin" > Number_of_columns = 26 > Number_of_rows = 453879 > Timestamp = 1313017647 > State = 1 > END HEADER > --snip-- > > thanks > > -- > Alan Kemp > email: alan <mailto:[email protected]>@three6five.com > <mailto:[email protected]> > mobile: +27 83 257 5970 > *three**6**five* systems > > > > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
