Hi Andrew, You could have each thread load up a bitvector per column. But then you would have to combine all those bitvectors into a single bitvector depending on using an AND or OR operation. This would have to use some sort of hierarchical merge (similar to a merge sort) to get the final bitvector. But then you could take the bit vector and use it to read each column in its own separate thread. So from that standpoint you'd get your parallelism. What could kill your parallelism though is as I say, multiple threads doing reads at the same time, and perhaps multiple threads doing memory allocs at the same time. If these resources are parallelizable then you should see some performance gain. Columns are independent entities and can be loaded separately. You might want to take a look at the table::select methods in the mensa.cpp file. They give you a good idea of how result sets are built and how you might be able to parallelize them...
Cheers, Teryl On Wed, Jul 13, 2011 at 1:35 PM, Olson, Andrew <[email protected]> wrote: > If you ignore the I/O bottleneck, can fastbit query evaluation be sped up > by using all of the cores in my server? I can envision different levels of > granularity for parallelization, but I anticipated using at most one thread > per column. Constraints on each column involved in the query could be > evaluated independently and then logical operations between columns could be > evaluated in parallel. > Andrew > > On Jul 13, 2011, at 12:06 PM, Teryl Taylor wrote: > > Hi Andrew, > > Not sure exactly what you mean. Are you talking about reading the > bitvectors of different columns each in its own thread? And then combining > them? Sort of how a merge sort works? Each column file index is separate > so there is no reason you wouldn't be able to do that. The only problem I > could foresee is if you have multiple threads reading different index files > from the same disk, your disk could become a bottleneck to slow things down. > Otherwise, doing it in parallel should work. I have been playing around > with the bitvector API quite a bit and I don't see any big gotchas. > > > Cheers, > > Teryl > > > On Wed, Jul 13, 2011 at 9:31 AM, Olson, Andrew <[email protected]<mailto: > [email protected]>> wrote: > I have a related question. Has anyone tried to do the bitwise operations > comparing bitvectors using multiple threads? This is more relevant for > queries involving MANY columns. > Andrew > > On Jul 12, 2011, at 10:46 PM, thiago wrote: > > Hello all, > as Teryl reported before, there is a -f flag that does the job. Is it me or > there isn't anything in the documentation about it? Just asking because I > believe this is an important information. > > Best, > Thiago > > > > On 12 July 2011 20:05, thiago <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto: > [email protected]>>> wrote: > > Hi, Teryl > Thanks for the reply, its good to know that. Today I implemented in > ibis.cpp a reading function. Now Ill try your suggestion. > > Best, > Thiago > > Em 12/07/2011 19:32, "Teryl Taylor" <[email protected]<mailto: > [email protected]><mailto:[email protected]<mailto: > [email protected]>>> escreveu: > > > Hi Thiago, > > > > Are you using the ibis program to do your querying? Just looking at the > > source code quickly, it looks like it has a "-f" parameter that can take > a > > file name with a select statement that ends with a ";". You could try > that. > > Have a look at the "readQueryFile" and "parse_args" methods in ibis.cpp > if > > you are unsure. > > > > HTH > > > > Cheers, > > > > Teryl > > > > > > > > On Tue, Jul 12, 2011 at 9:51 AM, thiago <[email protected] > <mailto:[email protected]><mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > >> Dear colleagues, > >> I am facing the following problem: my query is too long and it cannot be > >> passed through *-q* flag. Usually my queries have 300K characters and it > >> exceeds the shell limit itself. So I would like to know if there is any > way > >> I can pass the query string using an external file. > >> I wasn't able to find any clue in the documentation and I started to > >> consider ideas such as overcoming this restriction by some method or > even > >> implementing it in the source. Any help is appreciated, even some tips > on > >> how to search for this topic. > >> > >> Best regards, > >> Thiago > >> > >> > >> _______________________________________________ > >> FastBit-users mailing list > >> [email protected]<mailto:[email protected] > ><mailto:[email protected]<mailto:[email protected] > >> > >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > >> > >> > > _______________________________________________ > FastBit-users mailing list > [email protected]<mailto:[email protected]><mailto: > [email protected]<mailto:[email protected]>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ > FastBit-users mailing list > [email protected]<mailto:[email protected]> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ > FastBit-users mailing list > [email protected]<mailto:[email protected]> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >
_______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
