I wanted to a take moment to provide a status update to my experience with incorporating Fastbit into an iPad application. I am thoroughly impressed with how much better Fastbit performs than my sqlite-with-heavy-indexing strategy. Comparably, Fastbit has provided a 50x space improvement in the raw data table size including indices. The speed improvement varies depending on the specific queries, but in some cases I'm also seeing 50x speed improvement as well! In many cases I'm seeing at least 10x speed improvement.
John and I continue to work through a few issues that have been uncovered through some initial integration - I am very grateful for all of his help. I'm also thoroughly impressed with John's responsiveness and general helpfulness in assisting my adoption of the library. I'm optimistic that we'll soon have worked through all of the issues that are being uncovered through the integration process. Congrats on building such a great library and many thanks for your continued effort on making it even better. Michael On Fri, Aug 17, 2012 at 1:34 PM, Michael Beauregard <[email protected]> wrote: > Hi all, > > I'm building an analytical iOS application that currently uses sqlite to > slice and dice about 2.5MB (30K rows) of data. I've been able to get > performance to an acceptable level by relying upon many sql indices, one for > each slice/dice combination. Moreover, the queries would still be too slow > unless I ensure each query has its own "covering" index. This approach has > resulted in an explosion in the db size due to the vast indexing required > for performance. A 2.5MB dataset becomes 45MB when sufficiently indexed. > This strategy has taken us pretty far, but is at its performance and > scalability limits. > > My customer recently informed me that we now need to go well beyond (at > least 10x) our initial scale requirement. Plus he apparently needs to add > more ways to slice and dice the data, basically leading to further explosion > of the already unacceptable sql index size. This essentially means that I'm > not able to continue with my current architecture and I'm looking for other > options. > > I stumbled upon FastBit today while researching various options and I'm > really intrigued by what I see. However, I don't understand how I can > express the required queries because of FastBit's implicit group-by > functionality. Perhaps I'm just missing something obvious, but it seems like > I need explicit control over the group-by clause. > > For instance, here is a query that determines the number of events that > occurred in each month, broken down by month and priority of the event: > > SELECT IncidentBeginMonth, Priority, count(*) FROM events WHERE > IncidentBeginTime BETWEEN '2011-01-01 00:00:00' AND '2012-12-31 23:59:59' > GROUP BY IncidentBeginMonth, Priority ORDER BY IncidentBeginMonth, Priority > > The result is displayed in a stacked bar chart with one bar per month and > each bar broken down by event priority. This query executes in around 200ms > on 30K rows on an iPad 3 which is fast enough, but only barely. > > I would really appreciate if someone could help me figure out how to produce > an equivalent query in FastBit. I have already imported my data and have > been trying various queries using the ibis command, but I'm kind of stuck. > > Thanks for taking the time to read and consider my question. > > Michael > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
