Hi, Davey, Sorry about the silence. I have been working on a few other things and was unable to spend much time on fixing the problems with BLOB. Let me say that BLOB is not meant to participate in any kind of operations. We are only intending to support reading and writing of BLOBs.
When answering a query, the necessary data is read into memory. Depending on the specific query, what is necessary would be different. For string valued columns, the internally representation tries to stay with integers as much as possible. One possible complication here is in processing group-bys. For example, if a table is broken into two partitions, and the group-by operation involves strings, then when the two partitions have the same dictionary for the strings, the group-by operations can proceed in integer representation. On the other hand, if the two dictionaries for the columns with the same name are different, then the group-by operation has to be carried out with the strings explicitly. At this time, the group-by operation is carried out in memory. In other word, the data needed for the group-by operation including the dictionaries and other supporting information must fit in memory. This includes all intermediate values, not just the final results from the group-by operation. FastBit software works with data partition at a time. When a query involves 5 columns, then it is possible all 5 columns will have to be read into memory - exactly which columns are needed at the same time depends on the nature of the query. For example, if you have an arithmetic expression involve these 5 columns, then it would be necessary to have all of them in memory in order to evaluate the arithmetic expression. If the 5 columns are involved in 5 separate simple range conditions (such as "A > 5 and B < 6 ...") then most likely only the bitmap index associated with each column is partially read into memory - even in this case it is possible for FastBit to want to read the raw data if the raw data is relatively small compared to the corresponding index. Hop this helps.. John On 7/18/12 5:06 AM, Lidawei (Davey) wrote: > Hi John, > > Did I bother you? I have not receive your reply for a long time. > > I found BLOB support in following email list. Does BLOB support bitwise > operator which I can simulate IN() function, or will you realize it? > > Because my use case is in an important service, I need make everything under > control. So I want to known the memory usage of fastbit. Detail list > following: > > 1. Does fastbit read all rows which satisfied the condition? > > 2. Do only columns in select clause will hold in memory? > > 3. Does only a single part in memory at a certain time? If parts are > processed one by one. How you deal with order by? > > 4. For String column, is only position hold in memory? If so, is string > column query speed rely on the IO workload of File read. > > > > Best Regards, > Davey > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of > [email protected] > Sent: Tuesday, July 17, 2012 3:01 AM > To: [email protected] > Subject: FastBit-users Digest, Vol 59, Issue 13 > > Send FastBit-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of FastBit-users digest..." > > > Today's Topics: > > 1. Inserting binary (ibis::BLOB) data using > tablex::appendRow(const ibis::table::row&) > (Gerrit Hendrikus van Doorn) > 2. Re: Inserting binary (ibis::BLOB) data using > tablex::appendRow(const ibis::table::row&) (K. John Wu) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 15 Jul 2012 15:52:42 -0700 > From: Gerrit Hendrikus van Doorn <[email protected]> > Subject: [FastBit-users] Inserting binary (ibis::BLOB) data using > tablex::appendRow(const ibis::table::row&) > To: [email protected] > Message-ID: > <CAL_hiPpjhEHjhkS21fUWv5eqk-=qvpwcjp64jybvunbn48i...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > I've been using tablex::appendRow(const ibis::table::row&) to insert data > to a table. > I wanted to add some binary data using the ibis::BLOB type. When I tried > this I got a segmentation fault. Is the BLOB type supported by fastbit? I > did a quick scan through tafel.cpp and the appendRow method doesn't seem to > be doing anything with the ibis::BLOB type. Is there any other way to > insert binary data? > > Thanks, > Gerrit > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20120715/927b0f80/attachment-0001.htm > > > ------------------------------ > > Message: 2 > Date: Mon, 16 Jul 2012 07:43:00 -0700 > From: "K. John Wu" <[email protected]> > Subject: Re: [FastBit-users] Inserting binary (ibis::BLOB) data using > tablex::appendRow(const ibis::table::row&) > To: FastBit Users <[email protected]> > Cc: Gerrit Hendrikus van Doorn <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1 > > Dear Gerrit, > > We have been working on straightening out the code in dealing with > BLOBs. Unfortunately, it might still going to take a couple of weeks. > > John > > > On 7/15/12 3:52 PM, Gerrit Hendrikus van Doorn wrote: >> Hi, >> >> I've been using tablex::appendRow(const ibis::table::row&) to insert >> data to a table. >> I wanted to add some binary data using the ibis::BLOB type. When I >> tried this I got a segmentation fault. Is the BLOB type supported by >> fastbit? I did a quick scan through tafel.cpp and the appendRow method >> doesn't seem to be doing anything with the ibis::BLOB type. Is there >> any other way to insert binary data? >> >> Thanks, >> Gerrit >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > > ------------------------------ > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > End of FastBit-users Digest, Vol 59, Issue 13 > ********************************************* > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
