Hi again, To Jon: ======= As John mentioned we use the JNI interface, hence don't malloc ourselves. We also suspect that the problem is something to do with Java/Windows/32bit issues.
To John: ======== Still haven't had the time to run the test with the new code. Will do it, hopefully, tomorrow and post the results. We did run the test on Linux 64bit platform and the crash wasn't reproduced!! This leads us to believe that the cause to the problem is 32bit issues, and maybe Windows issues. We tend to let go and continue with our project, as in any case we plan to deploy on a 64bit Linux machine if required. (Still, I'll send the results on Windows as soon as I have them). To Olaf: ======== I've ran into your enhancement to FastBit's JNI and although it adds some nice abilities, we missed an interface for using ibis::mensa, because we want to query several partitions in a single select (we use mensa::select2 method. If there is a better/alternative way to achieve this, we'll be happy to hear about it). This is why we made some additions to capi.h and added JNI to them and didn't use your JNI. Thanks, Yoram Dayagi -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Olaf Walkowiak Sent: יום ג 15 פברואר 2011 09:56 To: FastBit Users Subject: Re: [FastBit-users] malloc fails with distinct count Hi John, he may want to try the alternative jni: https://bitbucket.org/olafW/fastbit4java BTW: Comments are welcome :-) Olaf Am 14.02.2011 17:35, schrieb K. John Wu: > Hi, Jon, > > Yoram is calling FrastBit from Java through our simple JNI interface. > The call to malloc is something inside FastBit. If there is a better > way, we will change the code. So, I hope you don't mind my intention > to seek some confirmation. > > Here are a couple of postings from comp.land.c, for example, > <http://groups.google.com/group/comp.lang.c/browse_thread/thread/91ac6d2a8ae91201/7eaa4d19cc39e863> > > and > <http://groups.google.com/group/comp.std.c/browse_thread/thread/22d575e7b5d36447/5d584b2b513eac88>. > > They seem to indicate that both of them return "logically" > contiguous memory. Are you talking about a different memory > allocation library? > > Potentially, there is some sort of interaction between Java virtual > machine and the JNI library. Does anyone have any experience with this? > > John > > > > On 2/14/2011 6:44 AM, Jon Strabala wrote: >> Yoram, >> >> Perhaps, the issue is related to a contiguous free memory block not >> being available. To clarify I have noted some of key the differences >> between malloc and calloc. I beleive if calloc can be used (if you >> are allocating arrays of data) your issue will go away. >> >> a. malloc takes a size input parameter of the memory block to be allocated >> b. malloc allocates memory as a single large contiguous block >> c. if a single contiguous block can't be be returned malloc fails >> >> a. calloc takes an input parameter for the number of blocks and an >> input parameter for the size of each block >> b. the memory allocated by calloc allocates may or may not be contiguous. >> c. all the blocks of memory are initialized to zero (0). >> d. unlike malloc, in the case of using calloc it is obvious that from >> b) above that calloc will not fail if the case memory can be allocated >> in multiple non-contiguous blocks when a single contiguous block >> allocation would fail. >> >> Regards, >> >> Jon Strabala >> >> On Sun, Feb 13, 2011 at 9:36 PM, K. John Wu<[email protected]> wrote: >>> Hi, Yoram, >>> >>> Thanks for your interest in FastBit. As far as I can the problem was >>> likely to be caused by a malloc problem. The puzzling part is this. >>> There apparently is enough memory on your machine (which appears to >>> have 4GB of RAM), and FastBit has set its own limit to 2GB. You were >>> using about 600MB when the latest malloc was issued. Apparently, that >>> malloc failed and FastBit went into a soft of clean up state. After >>> the clean up operations, there are still 500MB in memory. The malloc >>> was requesting 220MB. Therefore, this malloc should have succeeded. >>> But it failed. At which point, there was an exception thrown inside a >>> constructor which eventually led to a nil pointer to be generated from >>> the JNI code. >>> >>> There are a few speculations in the above. I have added a bit more >>> logging statements in FastBit code. If you check out the latest >>> source code from SVN repository<svn checkout >>> https://codeforge.lbl.gov/anonscm/fastbit>, and rerun the same test >>> case, you should be able to get a more detailed message. I am quite >>> curious what is the outcome. >>> >>> The real mystery is why malloc failed? I am not sure I have an good >>> answer. >>> >>> John >>> >>> >>> On 2/13/2011 5:05 AM, Yoram Dayagi wrote: >>>> Hi. >>>> >>>> I’m new to FastBit, evaluating it as an analytic database framework >>>> for our product. >>>> >>>> Our main use case is loading into memory portion of a rather large >>>> data set (3 billion rows per table) and then running thousands of >>>> queries on the data set loaded to memory. >>>> >>>> We develop in Java and use the FastBit integrated JNI implementation >>>> plus some additions to allow us to use ibis::mensa, as we want to >>>> query on several partitions together. >>>> >>>> So far we really love the framework, its features and performance. >>>> >>>> During our testing we encountered a crash when running the following: >>>> >>>> Configuration: >>>> >>>> -Windows 7 64bit >>>> >>>> -FastBit library compiled using Visual Studio 10 Express >>>> >>>> -Java 32bit (technical issues with JNI that prevents us from using 64bit) >>>> >>>> -A table with 40 partitions, each contains ~1.8M rows – total of ~72M >>>> rows (total of ~7.6GB on disk, out of them ~2.46GB of *.idx and *.msk >>>> files) >>>> >>>> -4 threads running random queries (select from a set of 100 queries) >>>> concurrently >>>> >>>> -All queries select ‘distinct(some column)’ using up to 6 equality >>>> conditions on the same 20 partitions >>>> >>>> -Running from eclipse >>>> >>>> After running some queries (about 130 queries from all threads >>>> together) the process crashed. According to the log it seems that the >>>> problem is memory allocation (see attached file. Note that some lines >>>> are cut from the middle of the file, as it is a very big one). >>>> >>>> Note that when the queries select ‘count(*)’ instead of ‘distinct(…)’ >>>> the process doesn’t crash. >>>> >>>> If anything is unclear or missing, I’ll be happy to provide as much >>>> information as possible in order to understand what is the cause for >>>> such behavior. >>>> >>>> Thanks, >>>> >>>> Yoram. >>>> >>>> >>>> Notice: The information contained in this e-mail message and/or >>>> attachments to it may contain confidential or privileged information. >>>> If you are not the intended recipient, any dissemination, use, review, >>>> distribution, printing or copying of the information contained in this >>>> e-mail message and/or attachments to it are strictly prohibited. If >>>> you have received this communication in error, please notify us by >>>> reply e-mail or telephone and immediately and permanently delete the >>>> message and any attachments. >>>> >>>> >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use,review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
