Dear FastBit users, I don't have any great ideas, but I attended Hadoop World NYC in October where there was a presentation on integrating Vertica and Hadoop. The slide show is at the following link. http://www.slideshare.net/cloudera/hw09-hadoop-vertica Vertica is a commercial column based database that is designed for analytics. They mention software they have developed for moving data between vertica and hadoop (map reduce). Vertica is used for rapid queries and map reduce for massive computations. There may be some ideas on Vertica's website for parallelizing FastBit in a more sophisticated manner.
Andrew On Dec 16, 2009, at 11:00 AM, K. John Wu wrote: > Dear Hyeongu Son, > > We are very glad that you are using FastBit. Regarding the use of > FastBit in a parallel environment, we have done a number of tests with > a relatively straightforward setup by having each processor working on > a different data partition. This may not be ideal if you have a large > number of machines in your cluster, but it has been shown to work > quite well and the user has to do only a minimal amount of programming. > > In order to do a more thorough integration with a system like Hadoop, > FastBit probably has to be put underneath the Hadoop run-time system. > As far as I know, HDFS does not support random I/O accesses, if this > is indeed the case, then it would require a lot of work to put FastBit > on top of HDFS -- something like writing a intermediate layer, say > based on FUSION, to translate random I/O accesses into HDFS calls. > > I am copying this message to the FastBit mailing list in hope of > someone else might have a better suggestion.. > > John > > > > On 12/16/2009 5:35 AM, Hyeongu Son wrote: >> Hello, this is Hyeongu Son who is Chungnam National Univ. in South Korea. >> I have a technical question >> Can I use the FastBit such as parallel DBMS and HDFS in cluster environment? >> I use HDFS(Hadoop Distributed File System), now. However, it is hard to >> employ data compression such as binary file. >> According to HDFS architecture, I can use it in serveral servers such as >> one file system. >> I read research papers and some document, but I could not find that >> FastBit is used in cluster system. >> How can I use FastBit in cluster system if it can be used? >> Thank you for reading my questions. >> your faithfully >> Hyeongu Son >> >> >> -- >> Hyeongu Son >> Office Room No. 424, >> Building No. Eng.2 >> Dept. of Compuer Science and Engineering, >> Chungnam National University >> Daejeon, Republic of Korea > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
