Hi, Aditya, Looks like you have run out of memory. I would suggest that you break up the data set into multiple partitions, where each partition has a subset of the rows. For a conservative estimate, you use the following,
8 bytes (per value, or whatever size is you actual data) * 200 columns * N rows If you have 16GB of memory, then you can have at most 10 million rows in memory at any given time. In this case, you might want to limit your data partition size to be 5 million rows. Good luck. John On 3/27/13 5:05 PM, Aditya Ramesh wrote: > Hi, > > When I try running a query on a large dataset using the ibis command > line tool, the query executes successfully. However, when I tried > using the JNI implementation from > https://bitbucket.org/olafW/fastbit4java/src, the query does not > succeed and it outputs a bunch of messages of the form: > Error -- fileManager::storage failed to malloc 1,423,308 bytes of > storage on retry > Error -- fileManager::storage failed to malloc 2,846,616 bytes of > storage on retry > Error -- bundles::ctor received an exception, start cleaning up > > Eventually, after some time, the program crashes completely with the message: > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 32744 bytes for > ChunkPool::allocate > # An error report file with more information is saved as: > # /export/home/eng/aramesh/LixCluster/hs_err_pid23388.log > > MyJVM args are "-server -Xms16384m -Xmx16384m -XX:PermSize=128M > -XX:MaxPermSize=128M -XX:NewSize=768m -XX:MaxNewSize=768m > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=85 > -XX:+AlwaysPreTouch -XX:+PrintGCApplicationStoppedTime > -XX:+PrintGCTimeStamps -XX:+UseCompressedOops > -XX:+ParallelRefProcEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps > -XX:+PrintTenuringDistribution -Xloggc:LixLogs/gc.log > -XX:ErrorFile=logs/hs_err.log -Djava.awt.headless=true > -Dcom.sun.management.jmxremote -XX:+HeapDumpOnOutOfMemoryError" > > The query is of the following form: SUM(m1), SUM(m2), .... SUM(m100) > where a = 'a1' and b = 'b1' (i.e, only one row should be returned > although there can be a large number of columns (from 60-200) to > retrieve). The query also only contains of conjunctions. > > Is there any way of resolving this problem with JNI? > > Thanks, > Aditya > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
