If you ignore the I/O bottleneck, can fastbit query evaluation be sped up by using all of the cores in my server? I can envision different levels of granularity for parallelization, but I anticipated using at most one thread per column. Constraints on each column involved in the query could be evaluated independently and then logical operations between columns could be evaluated in parallel. Andrew
On Jul 13, 2011, at 12:06 PM, Teryl Taylor wrote: Hi Andrew, Not sure exactly what you mean. Are you talking about reading the bitvectors of different columns each in its own thread? And then combining them? Sort of how a merge sort works? Each column file index is separate so there is no reason you wouldn't be able to do that. The only problem I could foresee is if you have multiple threads reading different index files from the same disk, your disk could become a bottleneck to slow things down. Otherwise, doing it in parallel should work. I have been playing around with the bitvector API quite a bit and I don't see any big gotchas. Cheers, Teryl On Wed, Jul 13, 2011 at 9:31 AM, Olson, Andrew <[email protected]<mailto:[email protected]>> wrote: I have a related question. Has anyone tried to do the bitwise operations comparing bitvectors using multiple threads? This is more relevant for queries involving MANY columns. Andrew On Jul 12, 2011, at 10:46 PM, thiago wrote: Hello all, as Teryl reported before, there is a -f flag that does the job. Is it me or there isn't anything in the documentation about it? Just asking because I believe this is an important information. Best, Thiago On 12 July 2011 20:05, thiago <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> wrote: Hi, Teryl Thanks for the reply, its good to know that. Today I implemented in ibis.cpp a reading function. Now Ill try your suggestion. Best, Thiago Em 12/07/2011 19:32, "Teryl Taylor" <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> escreveu: > Hi Thiago, > > Are you using the ibis program to do your querying? Just looking at the > source code quickly, it looks like it has a "-f" parameter that can take a > file name with a select statement that ends with a ";". You could try that. > Have a look at the "readQueryFile" and "parse_args" methods in ibis.cpp if > you are unsure. > > HTH > > Cheers, > > Teryl > > > > On Tue, Jul 12, 2011 at 9:51 AM, thiago > <[email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>>> > wrote: > >> Dear colleagues, >> I am facing the following problem: my query is too long and it cannot be >> passed through *-q* flag. Usually my queries have 300K characters and it >> exceeds the shell limit itself. So I would like to know if there is any way >> I can pass the query string using an external file. >> I wasn't able to find any clue in the documentation and I started to >> consider ideas such as overcoming this restriction by some method or even >> implementing it in the source. Any help is appreciated, even some tips on >> how to search for this topic. >> >> Best regards, >> Thiago >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> >> _______________________________________________ FastBit-users mailing list [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected]<mailto:[email protected]> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected]<mailto:[email protected]> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
