Hi, Yoram, Thanks for the feedback. The fact that the debug version got stuck is a good hint for us. We will look into this problem.
John On 2/17/2011 3:28 AM, Yoram Dayagi wrote: > Hi John, > I ran the test with the latest version of FastBit, as you requested. > Please find attached the log of this run. > Note that when the run finished, the process remained stuck and didn't exit > properly. I had to kill it. I don't think it has anything to do with the > crash (I created two fastbitjni.dll versions - one compiled with debug and > the other without debug. The debug version got stuck, while the release > version didn't get stuck. Both runs encountered the malloc problem, as > expected). > > Again, I must say that after testing the same code on Linux 64bit (which > passed successfully) we believe that this issue is not very urgent and is > probably caused by some combination of JNI and 32bit platform. > > Thanks, > > Yoram Dayagi > > > -----Original Message----- > From: Yoram Dayagi > Sent: יום ג 15 פברואר 2011 15:37 > To: FastBit Users > Subject: RE: [FastBit-users] malloc fails with distinct count > > Hi again, > > To Jon: > ======= > As John mentioned we use the JNI interface, hence don't malloc ourselves. We > also suspect that the problem is something to do with Java/Windows/32bit > issues. > > To John: > ======== > Still haven't had the time to run the test with the new code. Will do it, > hopefully, tomorrow and post the results. > We did run the test on Linux 64bit platform and the crash wasn't reproduced!! > This leads us to believe that the cause to the problem is 32bit issues, and > maybe Windows issues. We tend to let go and continue with our project, as in > any case we plan to deploy on a 64bit Linux machine if required. (Still, I'll > send the results on Windows as soon as I have them). > > To Olaf: > ======== > I've ran into your enhancement to FastBit's JNI and although it adds some > nice abilities, we missed an interface for using ibis::mensa, because we want > to query several partitions in a single select (we use mensa::select2 method. > If there is a better/alternative way to achieve this, we'll be happy to hear > about it). This is why we made some additions to capi.h and added JNI to them > and didn't use your JNI. > > Thanks, > Yoram Dayagi > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Olaf Walkowiak > Sent: יום ג 15 פברואר 2011 09:56 > To: FastBit Users > Subject: Re: [FastBit-users] malloc fails with distinct count > > Hi John, > > he may want to try the alternative jni: > https://bitbucket.org/olafW/fastbit4java > > BTW: Comments are welcome :-) > > Olaf > > > > Am 14.02.2011 17:35, schrieb K. John Wu: >> Hi, Jon, >> >> Yoram is calling FrastBit from Java through our simple JNI interface. >> The call to malloc is something inside FastBit. If there is a better >> way, we will change the code. So, I hope you don't mind my intention >> to seek some confirmation. >> >> Here are a couple of postings from comp.land.c, for example, >> <http://groups.google.com/group/comp.lang.c/browse_thread/thread/91ac6d2a8ae91201/7eaa4d19cc39e863> >> and >> <http://groups.google.com/group/comp.std.c/browse_thread/thread/22d575e7b5d36447/5d584b2b513eac88>. >> They seem to indicate that both of them return "logically" >> contiguous memory. Are you talking about a different memory >> allocation library? >> >> Potentially, there is some sort of interaction between Java virtual >> machine and the JNI library. Does anyone have any experience with this? >> >> John >> >> >> >> On 2/14/2011 6:44 AM, Jon Strabala wrote: >>> Yoram, >>> >>> Perhaps, the issue is related to a contiguous free memory block not >>> being available. To clarify I have noted some of key the differences >>> between malloc and calloc. I beleive if calloc can be used (if you >>> are allocating arrays of data) your issue will go away. >>> >>> a. malloc takes a size input parameter of the memory block to be allocated >>> b. malloc allocates memory as a single large contiguous block >>> c. if a single contiguous block can't be be returned malloc fails >>> >>> a. calloc takes an input parameter for the number of blocks and an >>> input parameter for the size of each block >>> b. the memory allocated by calloc allocates may or may not be contiguous. >>> c. all the blocks of memory are initialized to zero (0). >>> d. unlike malloc, in the case of using calloc it is obvious that from >>> b) above that calloc will not fail if the case memory can be allocated >>> in multiple non-contiguous blocks when a single contiguous block >>> allocation would fail. >>> >>> Regards, >>> >>> Jon Strabala >>> >>> On Sun, Feb 13, 2011 at 9:36 PM, K. John Wu<[email protected]> wrote: >>>> Hi, Yoram, >>>> >>>> Thanks for your interest in FastBit. As far as I can the problem was >>>> likely to be caused by a malloc problem. The puzzling part is this. >>>> There apparently is enough memory on your machine (which appears to >>>> have 4GB of RAM), and FastBit has set its own limit to 2GB. You were >>>> using about 600MB when the latest malloc was issued. Apparently, that >>>> malloc failed and FastBit went into a soft of clean up state. After >>>> the clean up operations, there are still 500MB in memory. The malloc >>>> was requesting 220MB. Therefore, this malloc should have succeeded. >>>> But it failed. At which point, there was an exception thrown inside a >>>> constructor which eventually led to a nil pointer to be generated from >>>> the JNI code. >>>> >>>> There are a few speculations in the above. I have added a bit more >>>> logging statements in FastBit code. If you check out the latest >>>> source code from SVN repository<svn checkout >>>> https://codeforge.lbl.gov/anonscm/fastbit>, and rerun the same test >>>> case, you should be able to get a more detailed message. I am quite >>>> curious what is the outcome. >>>> >>>> The real mystery is why malloc failed? I am not sure I have an good >>>> answer. >>>> >>>> John >>>> >>>> >>>> On 2/13/2011 5:05 AM, Yoram Dayagi wrote: >>>>> Hi. >>>>> >>>>> I’m new to FastBit, evaluating it as an analytic database framework >>>>> for our product. >>>>> >>>>> Our main use case is loading into memory portion of a rather large >>>>> data set (3 billion rows per table) and then running thousands of >>>>> queries on the data set loaded to memory. >>>>> >>>>> We develop in Java and use the FastBit integrated JNI implementation >>>>> plus some additions to allow us to use ibis::mensa, as we want to >>>>> query on several partitions together. >>>>> >>>>> So far we really love the framework, its features and performance. >>>>> >>>>> During our testing we encountered a crash when running the following: >>>>> >>>>> Configuration: >>>>> >>>>> -Windows 7 64bit >>>>> >>>>> -FastBit library compiled using Visual Studio 10 Express >>>>> >>>>> -Java 32bit (technical issues with JNI that prevents us from using 64bit) >>>>> >>>>> -A table with 40 partitions, each contains ~1.8M rows – total of ~72M >>>>> rows (total of ~7.6GB on disk, out of them ~2.46GB of *.idx and *.msk >>>>> files) >>>>> >>>>> -4 threads running random queries (select from a set of 100 queries) >>>>> concurrently >>>>> >>>>> -All queries select ‘distinct(some column)’ using up to 6 equality >>>>> conditions on the same 20 partitions >>>>> >>>>> -Running from eclipse >>>>> >>>>> After running some queries (about 130 queries from all threads >>>>> together) the process crashed. According to the log it seems that the >>>>> problem is memory allocation (see attached file. Note that some lines >>>>> are cut from the middle of the file, as it is a very big one). >>>>> >>>>> Note that when the queries select ‘count(*)’ instead of ‘distinct(…)’ >>>>> the process doesn’t crash. >>>>> >>>>> If anything is unclear or missing, I’ll be happy to provide as much >>>>> information as possible in order to understand what is the cause for >>>>> such behavior. >>>>> >>>>> Thanks, >>>>> >>>>> Yoram. >>>>> >>>>> >>>>> Notice: The information contained in this e-mail message and/or >>>>> attachments to it may contain confidential or privileged information. >>>>> If you are not the intended recipient, any dissemination, use, review, >>>>> distribution, printing or copying of the information contained in this >>>>> e-mail message and/or attachments to it are strictly prohibited. If >>>>> you have received this communication in error, please notify us by >>>>> reply e-mail or telephone and immediately and permanently delete the >>>>> message and any attachments. >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> FastBit-users mailing list >>>>> [email protected] >>>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > Notice: The information contained in this e-mail message and/or attachments > to it may contain confidential or privileged information. If you are not the > intended recipient, any dissemination, use,review, distribution, printing or > copying of the information contained in this e-mail message and/or > attachments to it are strictly prohibited. If you have received this > communication in error, please notify us by reply e-mail or telephone and > immediately and permanently delete the message and any attachments. > > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
