The hadoop release numbers have been notoriously hard to understand. Just because a release was made after another release, even if it has a higher number, doesn't mean it has all the features of the earlier release. Just look at: http://hadoop.apache.org/common/releases.html , 0.20.0 was released in 2009, followed by 0.19.2; later 0.21.0 was released, followed by several 0.20.x releases; then 0.23.0, followed by 0.22.0, and then 1.0.0. If that's not confusing, check out the "Download" portion of that page: http://hadoop.apache.org/common/releases.html#Download
1.0.x is the current stable version 1.1.x is the current beta version 0.23.x is the current alpha version The JIRA linked to below notes that it was fixed in version 0.21.0. Based on this chart: http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/ the 1.0.X releases are not based on the 0.21 branch. However, there does appear to be a 0.20-append branch that was merged in just prior to the 1.0 release. Though according to this: http://www.cloudera.com/blog/2009/07/file-appends-in-hdfs/ the history of append is... complicated. -Steven Willis > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Monday, May 14, 2012 1:30 PM > To: rsyslog-users > Subject: Re: [rsyslog] Append issues with omhdfs module > > It depends on what version of Hadoop and HDFS you're using. Alexis, > what distribution is being used? Stock Hadoop? CDH? > > On Mon, May 14, 2012 at 11:44 AM, Rainer Gerhards > <[email protected]> wrote: > > As far as I read this: > > > > https://issues.apache.org/jira/browse/HDFS-265 > > > > append seems to be there... > > > > Rainer > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Alexis Lelion > >> Sent: Monday, May 14, 2012 5:33 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Append issues with omhdfs module > >> > >> Well, the version I'm using is from February 2012 so released after > >> your HDFS support announcement :-) Hence why I was thinking about a > >> secret compile command or so, rather than a version issue. > >> But I'll investigate on Hadoop side, maybe this method was causing > >> issues and has been removed recently Thanks > >> > >> Alexis > >> > >> > >> On Mon, May 14, 2012 at 2:02 PM, Rainer Gerhards > >> <[email protected]> wrote: > >> > I remember that you need a pretty fresh version of hadoop. The > >> > append > >> method > >> > is vital for syslog logging, so I suggest you look at which > version > > supports > >> > it. > >> > > >> > HTH > >> > Rainer > >> > > >> >> -----Original Message----- > >> >> From: [email protected] [mailto:rsyslog- > >> >> [email protected]] On Behalf Of Alexis Lelion > >> >> Sent: Monday, May 14, 2012 1:31 PM > >> >> To: [email protected] > >> >> Subject: [rsyslog] Append issues with omhdfs module > >> >> > >> >> Dear, > >> >> > >> >> I'm currently building a hadoop cluster, and I was more than > happy > >> >> to see native support of HDFS in syslog, unfortunately I can't > get > >> >> it running. > >> >> My hadoop cluster works on top of debian squeeze + backports and > I > >> >> rebuilt the rsyslog package from the debian backports > >> >> (rsyslog-5.8.9) with omhdfs support. > >> >> Installation went fine, and everything launched without error. > But > >> >> when a piece of log has to be written in the hadoop cluster, I > get > >> >> the following error : > >> >> Exception in thread "Thread-3" java.io.IOException: Not supported > >> >> at > >> >> > >> > org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java: > >> >> 290) > >> >> at > >> >> org.apache.hadoop.fs.FileSystem.append(FileSystem.java:650) > >> >> Call to > >> >> > org.apache.hadoop.conf.FileSystem::append((Lorg/apache/hadoop/fs/P > >> >> ath;) > >> >> Lorg/apache/hadoop/fs/FSDataOutputStream;) > >> >> failed! > >> >> > >> >> So, not a syslog issue, but rather an hadoop issue. But when I > >> >> looked at hadoop code (stable version, 1.0.1), I got a bit > surprised : > >> >> 287 /** {@inheritDoc} */ > >> >> 288 public FSDataOutputStream append(Path f, int bufferSize, > >> >> 289 Progressable progress) throws IOException { > >> >> 290 throw new IOException("Not supported"); > >> >> 291 } > >> >> > >> >> This method has not been implemented yet... So my question is, > how > >> >> did you get syslog working with hadoop? Is there any Hadoop > >> >> setting I have to change to use another set of methods than this > 'append'? > >> >> Any help with this would be greatly appreciated Thanks! > >> >> > >> >> Alexis > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com/professional-services/ > >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
