Hi John, he may want to try the alternative jni: https://bitbucket.org/olafW/fastbit4java
BTW: Comments are welcome :-) Olaf Am 14.02.2011 17:35, schrieb K. John Wu: > Hi, Jon, > > Yoram is calling FrastBit from Java through our simple JNI interface. > The call to malloc is something inside FastBit. If there is a better > way, we will change the code. So, I hope you don't mind my intention > to seek some confirmation. > > Here are a couple of postings from comp.land.c, for example, > <http://groups.google.com/group/comp.lang.c/browse_thread/thread/91ac6d2a8ae91201/7eaa4d19cc39e863> > > and > <http://groups.google.com/group/comp.std.c/browse_thread/thread/22d575e7b5d36447/5d584b2b513eac88>. > > They seem to indicate that both of them return "logically" > contiguous memory. Are you talking about a different memory > allocation library? > > Potentially, there is some sort of interaction between Java virtual > machine and the JNI library. Does anyone have any experience with this? > > John > > > > On 2/14/2011 6:44 AM, Jon Strabala wrote: >> Yoram, >> >> Perhaps, the issue is related to a contiguous free memory block not >> being available. To clarify I have noted some of key the differences >> between malloc and calloc. I beleive if calloc can be used (if you >> are allocating arrays of data) your issue will go away. >> >> a. malloc takes a size input parameter of the memory block to be allocated >> b. malloc allocates memory as a single large contiguous block >> c. if a single contiguous block can't be be returned malloc fails >> >> a. calloc takes an input parameter for the number of blocks and an >> input parameter for the size of each block >> b. the memory allocated by calloc allocates may or may not be contiguous. >> c. all the blocks of memory are initialized to zero (0). >> d. unlike malloc, in the case of using calloc it is obvious that from >> b) above that calloc will not fail if the case memory can be allocated >> in multiple non-contiguous blocks when a single contiguous block >> allocation would fail. >> >> Regards, >> >> Jon Strabala >> >> On Sun, Feb 13, 2011 at 9:36 PM, K. John Wu<[email protected]> wrote: >>> Hi, Yoram, >>> >>> Thanks for your interest in FastBit. As far as I can the problem was >>> likely to be caused by a malloc problem. The puzzling part is this. >>> There apparently is enough memory on your machine (which appears to >>> have 4GB of RAM), and FastBit has set its own limit to 2GB. You were >>> using about 600MB when the latest malloc was issued. Apparently, that >>> malloc failed and FastBit went into a soft of clean up state. After >>> the clean up operations, there are still 500MB in memory. The malloc >>> was requesting 220MB. Therefore, this malloc should have succeeded. >>> But it failed. At which point, there was an exception thrown inside a >>> constructor which eventually led to a nil pointer to be generated from >>> the JNI code. >>> >>> There are a few speculations in the above. I have added a bit more >>> logging statements in FastBit code. If you check out the latest >>> source code from SVN repository<svn checkout >>> https://codeforge.lbl.gov/anonscm/fastbit>, and rerun the same test >>> case, you should be able to get a more detailed message. I am quite >>> curious what is the outcome. >>> >>> The real mystery is why malloc failed? I am not sure I have an good >>> answer. >>> >>> John >>> >>> >>> On 2/13/2011 5:05 AM, Yoram Dayagi wrote: >>>> Hi. >>>> >>>> I’m new to FastBit, evaluating it as an analytic database framework >>>> for our product. >>>> >>>> Our main use case is loading into memory portion of a rather large >>>> data set (3 billion rows per table) and then running thousands of >>>> queries on the data set loaded to memory. >>>> >>>> We develop in Java and use the FastBit integrated JNI implementation >>>> plus some additions to allow us to use ibis::mensa, as we want to >>>> query on several partitions together. >>>> >>>> So far we really love the framework, its features and performance. >>>> >>>> During our testing we encountered a crash when running the following: >>>> >>>> Configuration: >>>> >>>> -Windows 7 64bit >>>> >>>> -FastBit library compiled using Visual Studio 10 Express >>>> >>>> -Java 32bit (technical issues with JNI that prevents us from using 64bit) >>>> >>>> -A table with 40 partitions, each contains ~1.8M rows – total of ~72M >>>> rows (total of ~7.6GB on disk, out of them ~2.46GB of *.idx and *.msk >>>> files) >>>> >>>> -4 threads running random queries (select from a set of 100 queries) >>>> concurrently >>>> >>>> -All queries select ‘distinct(some column)’ using up to 6 equality >>>> conditions on the same 20 partitions >>>> >>>> -Running from eclipse >>>> >>>> After running some queries (about 130 queries from all threads >>>> together) the process crashed. According to the log it seems that the >>>> problem is memory allocation (see attached file. Note that some lines >>>> are cut from the middle of the file, as it is a very big one). >>>> >>>> Note that when the queries select ‘count(*)’ instead of ‘distinct(…)’ >>>> the process doesn’t crash. >>>> >>>> If anything is unclear or missing, I’ll be happy to provide as much >>>> information as possible in order to understand what is the cause for >>>> such behavior. >>>> >>>> Thanks, >>>> >>>> Yoram. >>>> >>>> >>>> Notice: The information contained in this e-mail message and/or >>>> attachments to it may contain confidential or privileged information. >>>> If you are not the intended recipient, any dissemination, use, review, >>>> distribution, printing or copying of the information contained in this >>>> e-mail message and/or attachments to it are strictly prohibited. If >>>> you have received this communication in error, please notify us by >>>> reply e-mail or telephone and immediately and permanently delete the >>>> message and any attachments. >>>> >>>> >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
