It depends on what version of Hadoop and HDFS you're using. Alexis, what distribution is being used? Stock Hadoop? CDH?
On Mon, May 14, 2012 at 11:44 AM, Rainer Gerhards <[email protected]> wrote: > As far as I read this: > > https://issues.apache.org/jira/browse/HDFS-265 > > append seems to be there... > > Rainer > >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Alexis Lelion >> Sent: Monday, May 14, 2012 5:33 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Append issues with omhdfs module >> >> Well, the version I'm using is from February 2012 so released after >> your HDFS support announcement :-) >> Hence why I was thinking about a secret compile command or so, rather >> than a version issue. >> But I'll investigate on Hadoop side, maybe this method was causing >> issues and has been removed recently >> Thanks >> >> Alexis >> >> >> On Mon, May 14, 2012 at 2:02 PM, Rainer Gerhards >> <[email protected]> wrote: >> > I remember that you need a pretty fresh version of hadoop. The append >> method >> > is vital for syslog logging, so I suggest you look at which version > supports >> > it. >> > >> > HTH >> > Rainer >> > >> >> -----Original Message----- >> >> From: [email protected] [mailto:rsyslog- >> >> [email protected]] On Behalf Of Alexis Lelion >> >> Sent: Monday, May 14, 2012 1:31 PM >> >> To: [email protected] >> >> Subject: [rsyslog] Append issues with omhdfs module >> >> >> >> Dear, >> >> >> >> I'm currently building a hadoop cluster, and I was more than happy to >> >> see native support of HDFS in syslog, unfortunately I can't get it >> >> running. >> >> My hadoop cluster works on top of debian squeeze + backports and I >> >> rebuilt the rsyslog package from the debian backports (rsyslog-5.8.9) >> >> with omhdfs support. >> >> Installation went fine, and everything launched without error. But >> >> when a piece of log has to be written in the hadoop cluster, I get the >> >> following error : >> >> Exception in thread "Thread-3" java.io.IOException: Not supported >> >> at >> >> >> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java: >> >> 290) >> >> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:650) >> >> Call to >> >> org.apache.hadoop.conf.FileSystem::append((Lorg/apache/hadoop/fs/Path;) >> >> Lorg/apache/hadoop/fs/FSDataOutputStream;) >> >> failed! >> >> >> >> So, not a syslog issue, but rather an hadoop issue. But when I looked >> >> at hadoop code (stable version, 1.0.1), I got a bit surprised : >> >> 287 /** {@inheritDoc} */ >> >> 288 public FSDataOutputStream append(Path f, int bufferSize, >> >> 289 Progressable progress) throws IOException { >> >> 290 throw new IOException("Not supported"); >> >> 291 } >> >> >> >> This method has not been implemented yet... So my question is, how did >> >> you get syslog working with hadoop? Is there any Hadoop setting I have >> >> to change to use another set of methods than this 'append'? >> >> Any help with this would be greatly appreciated >> >> Thanks! >> >> >> >> Alexis >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
