Hi, Yes, HDFS does not allow random I/O access. But what about HBase (a column-oriented database that reads and writes data to/from HDFS)? Maybe there is room for Fastbit there!
Otis -- Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch ----- Original Message ---- > From: K. John Wu <[email protected]> > To: FastBit Users <[email protected]> > Sent: Thu, December 17, 2009 4:42:38 PM > Subject: Re: [FastBit-users] Technical question > > Hi, Andrew, > > Thanks for the information. For those who are interested in reading > more about hadoop and vertica, there are more information at > . > > In the same vain, data warehousing vendors like aster and greenplum > are also trying to take advantage of mapreduce paradigm as well, see > and > . > > Here is an interesting article that lists who else is latching on to > the same thing > > > John > > > On 12/16/2009 12:42 PM, Andrew Olson wrote: > > Dear FastBit users, > > I don't have any great ideas, but I attended Hadoop World NYC in October > > where > there was a presentation on integrating Vertica and Hadoop. The slide show > is > at the following link. > > http://www.slideshare.net/cloudera/hw09-hadoop-vertica > > Vertica is a commercial column based database that is designed for > > analytics. > They mention software they have developed for moving data between vertica and > hadoop (map reduce). Vertica is used for rapid queries and map reduce for > massive computations. There may be some ideas on Vertica's website for > parallelizing FastBit in a more sophisticated manner. > > > > Andrew > > > > On Dec 16, 2009, at 11:00 AM, K. John Wu wrote: > > > >> Dear Hyeongu Son, > >> > >> We are very glad that you are using FastBit. Regarding the use of > >> FastBit in a parallel environment, we have done a number of tests with > >> a relatively straightforward setup by having each processor working on > >> a different data partition. This may not be ideal if you have a large > >> number of machines in your cluster, but it has been shown to work > >> quite well and the user has to do only a minimal amount of programming. > >> > >> In order to do a more thorough integration with a system like Hadoop, > >> FastBit probably has to be put underneath the Hadoop run-time system. > >> As far as I know, HDFS does not support random I/O accesses, if this > >> is indeed the case, then it would require a lot of work to put FastBit > >> on top of HDFS -- something like writing a intermediate layer, say > >> based on FUSION, to translate random I/O accesses into HDFS calls. > >> > >> I am copying this message to the FastBit mailing list in hope of > >> someone else might have a better suggestion.. > >> > >> John > >> > >> > >> > >> On 12/16/2009 5:35 AM, Hyeongu Son wrote: > >>> Hello, this is Hyeongu Son who is Chungnam National Univ. in South Korea. > >>> I have a technical question > >>> Can I use the FastBit such as parallel DBMS and HDFS in cluster > >>> environment? > >>> I use HDFS(Hadoop Distributed File System), now. However, it is hard to > >>> employ data compression such as binary file. > >>> According to HDFS architecture, I can use it in serveral servers such as > >>> one file system. > >>> I read research papers and some document, but I could not find that > >>> FastBit is used in cluster system. > >>> How can I use FastBit in cluster system if it can be used? > >>> Thank you for reading my questions. > >>> your faithfully > >>> Hyeongu Son > >>> > >>> > >>> -- > >>> Hyeongu Son > >>> Office Room No. 424, > >>> Building No. Eng.2 > >>> Dept. of Compuer Science and Engineering, > >>> Chungnam National University > >>> Daejeon, Republic of Korea > >> _______________________________________________ > >> FastBit-users mailing list > >> [email protected] > >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > _______________________________________________ > > FastBit-users mailing list > > [email protected] > > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
