Well, yes and no. Yes in terms it'd be possible, but no because if you want the scalability of scribe you'd have to run a broker on every (other) node, which is rather complex.
Scribe is, in practice, a very simple Thrift RPC service serving requests (asynchronously) , without persistence guarantees and focused on throughput towards later batch processing (hadoop + hive/pig typically). Qpid/AMQP is not really an obvious fit for logging in the sense of a syslog replacement, but it's definitely an excellent choice for application process and audit logging - where persistence and routing capabilities are much more interesting than throughput. Now, that said, if I were to implement high throughput logging over AMQP I'd probably do preliminary classification and aggregation of log messages locally on the collecting node, resulting in bundles (hundreds) of log records by routing predicate / subject. Sending multiple log records per AMQP message would ensure you're more or less io bound regardless of number of producers/nodes. If you're just doing low-volume logging though, just go ahead and publish an AMQP message for each log record, and rely on qpid persistence, flow control and routing. /d On Fri, Oct 29, 2010 at 3:00 AM, Olivier Devaux <[email protected]> wrote: > Hello, > > I'm new to QPID and looking at some log transport tools. > It looks like QPID could be used like Facebook's Scribe but would bring more > flexibility to root different types of logs. > > Was this already done ? What do you think ? > > Thanks > > Oliv > --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
