These days I'd assume it would be simpler to use Apache Flume Syslog Source or omkafka to write from rsyslog to Kafka. Once Flume has the data, one can use HDFS, Hive, and other Sinks that are being actively developed and maintained. Similarly, there are lots of tools for getting data from Kafka to HDFS, like LinkedIn's Camus for instance.
Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Feb 26, 2015 at 10:15 AM, Rainer Gerhards <[email protected]> wrote: > 2015-02-25 19:50 GMT+01:00 Kendall Green <[email protected]>: > > > Please can anyone help clarify how-to output to hadoop from rsyslog? > > > > omhdfs produces the following error on start with version rsyslog 8.8.0 > > > > (unable to get stack trace for java.lang.NoClassDefFoundError exception: > > ExceptionUtils::getStackTrace error.) > > hdfsBuilderConnect(forceNewInstance=0, nn=10.0.0.10, port=8020, > > kerbTicketCachePath=(NULL), userName-(NULL)) error: > > (unable to get stack trace for java.lang.NoClassDefFoundError exception: > > ExceptionUtils::getStackTrace error.) > > > > > > - error log when starting rsyslogd: > > rsyslogd-2179: omdhfs: failed to open /rsyslog/omhdfs_test01.txt - > retrying > > later [try http://www.rsyslog.com/e/2179 ] > > > > > > The cloudera hadoop setup working with hdfs dfs commands to access, read, > > write, and delete data from hadoop without authentication. However the > > omhdfs module does not appear to work for rsyslog v8. > > > > Produces same results with or without the fix for unsigned type argument > in > > omhdfs.c that was reported in a previous message list post. The end > result > > from testing the module which seems to build successfully but in use > > produces the same errors. It is interesting that kerbTicketCachePath, > extra > > userName parameter appears in error output, although is not a > configuration > > parameter for omhdfs. > > > > Does anyone has experience with omhdfs module? > > > > > A bit of history. omhdfs was done as sponsored work, and I should really > move it to the contrib module tree. The problem is that it is a real mess > to get the Java interface going AND it is an even greater mess to find a > version that is compatible with the feature set we need ("append" > functionality specifically). At the time I wrote the module, the "file > append" functionality was scheduled to become part of the next mainstream > release (at least I was told so at that time). As it turned out, this did > not happen for quite a while and I don't know if it finally is possible of > not. Given the fact that it is an module with very low interest, I did not > put my effort into it but rather things more often requested. > > I think, though, that there was a recent ML thread with a solution. I need > to follow that up as well, and will do as soon as time permits. If that's a > permanent solution, I'll follow that adivse. > > Rainer > > > Thanks! > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
