Ari, Thanks for the input. I found another thread on the general hadoop list that pertains to the the discussion. I'll post here for future readers:
http://mail-archives.apache.org/mod_mbox/hadoop-general/200911.mbox/%3cc7397f6e.53ea%25ey...@yahoo-inc.com%3e Your explanation helps a lot though. I'd second the idea of have a Chukwa-vs page on the website to help people understand what it can do for them. We are trying to get our logs into HDFS so I think chukwa will provide a lot of value in this regard. -- Corbin On Dec 24, 2009, at 9:51 AM, Ariel Rabkin wrote: > Howdy. > > It's a good question, and maybe we should have a > Chukwa-vs-the-competition page on the website. > > Chukwa is designed to help collect system logs, and store them in > HDFS. Scribe is designed as more of a syslog++. We went to some > trouble to make sure our metadata and output formats were MapReduce > friendly, whereas I think they leave that much more up to you. > > My understanding is that Scribe exposes a Thrift service, only. So if > you want to use Scribe, you pretty much have to go all-in and modify > your app and deployment to send logs via Thrift. Chukwa can just read > whatever files or data sources are lying around. We can pretty easily > listen for incoming Thrift RPC calls -- I believe Jerome is currently > coding that up. > > Chukwa comes with visualization and processing tools, whereas Scribe doesn't. > > Scribe might be slightly lower overhead -- I believe it's C++ and > Chukwa is Java. Chukwa isn't too big, either, so the absolute > difference might not be significant. > > --Ari > > On Wed, Dec 23, 2009 at 3:06 PM, Corbin Hoenes <cor...@tynt.com> wrote: >> Does anyone know of a comparison between Chukwa and Scribe( >> http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tutorials >> ) >> I am interested in why someone might choose Chukwa vs. Scribe and their >> pros/cons. >> >> >> > > > > -- > Ari Rabkin asrab...@gmail.com > UC Berkeley Computer Science Department Corbin Hoenes cor...@tynt.com skype: choenes
