Hi, Jon, Yoram is calling FrastBit from Java through our simple JNI interface. The call to malloc is something inside FastBit. If there is a better way, we will change the code. So, I hope you don't mind my intention to seek some confirmation.
Here are a couple of postings from comp.land.c, for example, <http://groups.google.com/group/comp.lang.c/browse_thread/thread/91ac6d2a8ae91201/7eaa4d19cc39e863> and <http://groups.google.com/group/comp.std.c/browse_thread/thread/22d575e7b5d36447/5d584b2b513eac88>. They seem to indicate that both of them return "logically" contiguous memory. Are you talking about a different memory allocation library? Potentially, there is some sort of interaction between Java virtual machine and the JNI library. Does anyone have any experience with this? John On 2/14/2011 6:44 AM, Jon Strabala wrote: > Yoram, > > Perhaps, the issue is related to a contiguous free memory block not > being available. To clarify I have noted some of key the differences > between malloc and calloc. I beleive if calloc can be used (if you > are allocating arrays of data) your issue will go away. > > a. malloc takes a size input parameter of the memory block to be allocated > b. malloc allocates memory as a single large contiguous block > c. if a single contiguous block can't be be returned malloc fails > > a. calloc takes an input parameter for the number of blocks and an > input parameter for the size of each block > b. the memory allocated by calloc allocates may or may not be contiguous. > c. all the blocks of memory are initialized to zero (0). > d. unlike malloc, in the case of using calloc it is obvious that from > b) above that calloc will not fail if the case memory can be allocated > in multiple non-contiguous blocks when a single contiguous block > allocation would fail. > > Regards, > > Jon Strabala > > On Sun, Feb 13, 2011 at 9:36 PM, K. John Wu<[email protected]> wrote: >> Hi, Yoram, >> >> Thanks for your interest in FastBit. As far as I can the problem was >> likely to be caused by a malloc problem. The puzzling part is this. >> There apparently is enough memory on your machine (which appears to >> have 4GB of RAM), and FastBit has set its own limit to 2GB. You were >> using about 600MB when the latest malloc was issued. Apparently, that >> malloc failed and FastBit went into a soft of clean up state. After >> the clean up operations, there are still 500MB in memory. The malloc >> was requesting 220MB. Therefore, this malloc should have succeeded. >> But it failed. At which point, there was an exception thrown inside a >> constructor which eventually led to a nil pointer to be generated from >> the JNI code. >> >> There are a few speculations in the above. I have added a bit more >> logging statements in FastBit code. If you check out the latest >> source code from SVN repository<svn checkout >> https://codeforge.lbl.gov/anonscm/fastbit>, and rerun the same test >> case, you should be able to get a more detailed message. I am quite >> curious what is the outcome. >> >> The real mystery is why malloc failed? I am not sure I have an good >> answer. >> >> John >> >> >> On 2/13/2011 5:05 AM, Yoram Dayagi wrote: >>> Hi. >>> >>> I’m new to FastBit, evaluating it as an analytic database framework >>> for our product. >>> >>> Our main use case is loading into memory portion of a rather large >>> data set (3 billion rows per table) and then running thousands of >>> queries on the data set loaded to memory. >>> >>> We develop in Java and use the FastBit integrated JNI implementation >>> plus some additions to allow us to use ibis::mensa, as we want to >>> query on several partitions together. >>> >>> So far we really love the framework, its features and performance. >>> >>> During our testing we encountered a crash when running the following: >>> >>> Configuration: >>> >>> -Windows 7 64bit >>> >>> -FastBit library compiled using Visual Studio 10 Express >>> >>> -Java 32bit (technical issues with JNI that prevents us from using 64bit) >>> >>> -A table with 40 partitions, each contains ~1.8M rows – total of ~72M >>> rows (total of ~7.6GB on disk, out of them ~2.46GB of *.idx and *.msk >>> files) >>> >>> -4 threads running random queries (select from a set of 100 queries) >>> concurrently >>> >>> -All queries select ‘distinct(some column)’ using up to 6 equality >>> conditions on the same 20 partitions >>> >>> -Running from eclipse >>> >>> After running some queries (about 130 queries from all threads >>> together) the process crashed. According to the log it seems that the >>> problem is memory allocation (see attached file. Note that some lines >>> are cut from the middle of the file, as it is a very big one). >>> >>> Note that when the queries select ‘count(*)’ instead of ‘distinct(…)’ >>> the process doesn’t crash. >>> >>> If anything is unclear or missing, I’ll be happy to provide as much >>> information as possible in order to understand what is the cause for >>> such behavior. >>> >>> Thanks, >>> >>> Yoram. >>> >>> >>> Notice: The information contained in this e-mail message and/or >>> attachments to it may contain confidential or privileged information. >>> If you are not the intended recipient, any dissemination, use, review, >>> distribution, printing or copying of the information contained in this >>> e-mail message and/or attachments to it are strictly prohibited. If >>> you have received this communication in error, please notify us by >>> reply e-mail or telephone and immediately and permanently delete the >>> message and any attachments. >>> >>> >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
