Hi all, I built hadoop using the stable sources from apache, so without append support apparently. I will give a try to a beta or alpha release supporting it. Thanks for the doc about append Steven, it was really interesting.
On Mon, May 14, 2012 at 8:40 PM, Steven Willis <[email protected]> wrote: > The hadoop release numbers have been notoriously hard to understand. Just > because a release was made after another release, even if it has a higher > number, doesn't mean it has all the features of the earlier release. Just > look at: http://hadoop.apache.org/common/releases.html , 0.20.0 was released > in 2009, followed by 0.19.2; later 0.21.0 was released, followed by several > 0.20.x releases; then 0.23.0, followed by 0.22.0, and then 1.0.0. If that's > not confusing, check out the "Download" portion of that page: > http://hadoop.apache.org/common/releases.html#Download > > 1.0.x is the current stable version > 1.1.x is the current beta version > 0.23.x is the current alpha version > > The JIRA linked to below notes that it was fixed in version 0.21.0. Based on > this chart: > http://www.cloudera.com/blog/2012/01/an-update-on-apache-hadoop-1-0/ the > 1.0.X releases are not based on the 0.21 branch. However, there does appear > to be a 0.20-append branch that was merged in just prior to the 1.0 release. > Though according to this: > http://www.cloudera.com/blog/2009/07/file-appends-in-hdfs/ the history of > append is... complicated. > > -Steven Willis > >> -----Original Message----- >> From: [email protected] [mailto:rsyslog- >> [email protected]] On Behalf Of Aaron Wiebe >> Sent: Monday, May 14, 2012 1:30 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Append issues with omhdfs module >> >> It depends on what version of Hadoop and HDFS you're using. Alexis, >> what distribution is being used? Stock Hadoop? CDH? >> >> On Mon, May 14, 2012 at 11:44 AM, Rainer Gerhards >> <[email protected]> wrote: >> > As far as I read this: >> > >> > https://issues.apache.org/jira/browse/HDFS-265 >> > >> > append seems to be there... >> > >> > Rainer >> > >> >> -----Original Message----- >> >> From: [email protected] [mailto:rsyslog- >> >> [email protected]] On Behalf Of Alexis Lelion >> >> Sent: Monday, May 14, 2012 5:33 PM >> >> To: rsyslog-users >> >> Subject: Re: [rsyslog] Append issues with omhdfs module >> >> >> >> Well, the version I'm using is from February 2012 so released after >> >> your HDFS support announcement :-) Hence why I was thinking about a >> >> secret compile command or so, rather than a version issue. >> >> But I'll investigate on Hadoop side, maybe this method was causing >> >> issues and has been removed recently Thanks >> >> >> >> Alexis >> >> >> >> >> >> On Mon, May 14, 2012 at 2:02 PM, Rainer Gerhards >> >> <[email protected]> wrote: >> >> > I remember that you need a pretty fresh version of hadoop. The >> >> > append >> >> method >> >> > is vital for syslog logging, so I suggest you look at which >> version >> > supports >> >> > it. >> >> > >> >> > HTH >> >> > Rainer >> >> > >> >> >> -----Original Message----- >> >> >> From: [email protected] [mailto:rsyslog- >> >> >> [email protected]] On Behalf Of Alexis Lelion >> >> >> Sent: Monday, May 14, 2012 1:31 PM >> >> >> To: [email protected] >> >> >> Subject: [rsyslog] Append issues with omhdfs module >> >> >> >> >> >> Dear, >> >> >> >> >> >> I'm currently building a hadoop cluster, and I was more than >> happy >> >> >> to see native support of HDFS in syslog, unfortunately I can't >> get >> >> >> it running. >> >> >> My hadoop cluster works on top of debian squeeze + backports and >> I >> >> >> rebuilt the rsyslog package from the debian backports >> >> >> (rsyslog-5.8.9) with omhdfs support. >> >> >> Installation went fine, and everything launched without error. >> But >> >> >> when a piece of log has to be written in the hadoop cluster, I >> get >> >> >> the following error : >> >> >> Exception in thread "Thread-3" java.io.IOException: Not supported >> >> >> at >> >> >> >> >> >> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java: >> >> >> 290) >> >> >> at >> >> >> org.apache.hadoop.fs.FileSystem.append(FileSystem.java:650) >> >> >> Call to >> >> >> >> org.apache.hadoop.conf.FileSystem::append((Lorg/apache/hadoop/fs/P >> >> >> ath;) >> >> >> Lorg/apache/hadoop/fs/FSDataOutputStream;) >> >> >> failed! >> >> >> >> >> >> So, not a syslog issue, but rather an hadoop issue. But when I >> >> >> looked at hadoop code (stable version, 1.0.1), I got a bit >> surprised : >> >> >> 287 /** {@inheritDoc} */ >> >> >> 288 public FSDataOutputStream append(Path f, int bufferSize, >> >> >> 289 Progressable progress) throws IOException { >> >> >> 290 throw new IOException("Not supported"); >> >> >> 291 } >> >> >> >> >> >> This method has not been implemented yet... So my question is, >> how >> >> >> did you get syslog working with hadoop? Is there any Hadoop >> >> >> setting I have to change to use another set of methods than this >> 'append'? >> >> >> Any help with this would be greatly appreciated Thanks! >> >> >> >> >> >> Alexis >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
