Hi, Mark, Thanks for your interest in FastBit software. Looks like you are talking about number of columns in a query expression. In this case, the practical limitation would be memory required to hold the columns in core. A rough estimate goes like this, if your query selects 1 million rows (before group-by operations) and 100 columns, then you should be prepared to store 100 million values in memory. For some group by operations, a second copy of the data is generated, which can double the space requirement.
If your group by operations can be processed one data partition at a time, less memory might be needed, but the above back of the envelop number is useful to keep in mind. John On 5/30/13 11:47 AM, Mark Hansen wrote: > I'm wondering if there is any practical limit on the number of columns > in a FastBit "table"? > > My company is looking at FastBit as a potential backend for an > analytics engine. We would have one big table with about 100 columns > - 30 of which need to be indexed for searching. We call these the > "dimensions". The other 70 - the "metrics" - do not need to be > indexed, but might need some aggregation like SUM, AVG, STDEV, etc. > The data is going to be pretty sparse - typically 3-4 "dimensions" are > populated and 8-12 "metrics" per row, with the rest of the columns set > to NULL. > > Queries will be typically like: > > SELECT d1, d5, d18, m2, m19, m29 FROM TABLE where d1 between > lowerBound and upperBound > > We are looking at about 100 columns and 100 MILLION rows. A few of > the dimension columns (that need to be indexed) contain arbitrary > length text values. > > We could partition the table and decrease the rows to about 10 MILLION > if necessary. > > Does this sound like something that FastBit is well suited for? > > -- Mark > > -- > Mark Hansen > Founder & President > Digital Brand Mine | 708 3rd Ave | New York, New York 10017 > office: 212-961-7250 > cell: 914-924-3398 > http://digitalbrandmine.com/ | email: [email protected] > <mailto:[email protected]> > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
