Yoram, Perhaps, the issue is related to a contiguous free memory block not being available. To clarify I have noted some of key the differences between malloc and calloc. I beleive if calloc can be used (if you are allocating arrays of data) your issue will go away.
a. malloc takes a size input parameter of the memory block to be allocated b. malloc allocates memory as a single large contiguous block c. if a single contiguous block can't be be returned malloc fails a. calloc takes an input parameter for the number of blocks and an input parameter for the size of each block b. the memory allocated by calloc allocates may or may not be contiguous. c. all the blocks of memory are initialized to zero (0). d. unlike malloc, in the case of using calloc it is obvious that from b) above that calloc will not fail if the case memory can be allocated in multiple non-contiguous blocks when a single contiguous block allocation would fail. Regards, Jon Strabala On Sun, Feb 13, 2011 at 9:36 PM, K. John Wu <[email protected]> wrote: > Hi, Yoram, > > Thanks for your interest in FastBit. As far as I can the problem was > likely to be caused by a malloc problem. The puzzling part is this. > There apparently is enough memory on your machine (which appears to > have 4GB of RAM), and FastBit has set its own limit to 2GB. You were > using about 600MB when the latest malloc was issued. Apparently, that > malloc failed and FastBit went into a soft of clean up state. After > the clean up operations, there are still 500MB in memory. The malloc > was requesting 220MB. Therefore, this malloc should have succeeded. > But it failed. At which point, there was an exception thrown inside a > constructor which eventually led to a nil pointer to be generated from > the JNI code. > > There are a few speculations in the above. I have added a bit more > logging statements in FastBit code. If you check out the latest > source code from SVN repository <svn checkout > https://codeforge.lbl.gov/anonscm/fastbit>, and rerun the same test > case, you should be able to get a more detailed message. I am quite > curious what is the outcome. > > The real mystery is why malloc failed? I am not sure I have an good > answer. > > John > > > On 2/13/2011 5:05 AM, Yoram Dayagi wrote: >> Hi. >> >> I’m new to FastBit, evaluating it as an analytic database framework >> for our product. >> >> Our main use case is loading into memory portion of a rather large >> data set (3 billion rows per table) and then running thousands of >> queries on the data set loaded to memory. >> >> We develop in Java and use the FastBit integrated JNI implementation >> plus some additions to allow us to use ibis::mensa, as we want to >> query on several partitions together. >> >> So far we really love the framework, its features and performance. >> >> During our testing we encountered a crash when running the following: >> >> Configuration: >> >> -Windows 7 64bit >> >> -FastBit library compiled using Visual Studio 10 Express >> >> -Java 32bit (technical issues with JNI that prevents us from using 64bit) >> >> -A table with 40 partitions, each contains ~1.8M rows – total of ~72M >> rows (total of ~7.6GB on disk, out of them ~2.46GB of *.idx and *.msk >> files) >> >> -4 threads running random queries (select from a set of 100 queries) >> concurrently >> >> -All queries select ‘distinct(some column)’ using up to 6 equality >> conditions on the same 20 partitions >> >> -Running from eclipse >> >> After running some queries (about 130 queries from all threads >> together) the process crashed. According to the log it seems that the >> problem is memory allocation (see attached file. Note that some lines >> are cut from the middle of the file, as it is a very big one). >> >> Note that when the queries select ‘count(*)’ instead of ‘distinct(…)’ >> the process doesn’t crash. >> >> If anything is unclear or missing, I’ll be happy to provide as much >> information as possible in order to understand what is the cause for >> such behavior. >> >> Thanks, >> >> Yoram. >> >> >> Notice: The information contained in this e-mail message and/or >> attachments to it may contain confidential or privileged information. >> If you are not the intended recipient, any dissemination, use, review, >> distribution, printing or copying of the information contained in this >> e-mail message and/or attachments to it are strictly prohibited. If >> you have received this communication in error, please notify us by >> reply e-mail or telephone and immediately and permanently delete the >> message and any attachments. >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
