I put a breakpoint and called close() on the HdfsOutputStream (via the display window in eclipse) and then file was created (correctly named and with the expected contents) so it appears that the issue is with either a non-terminating process resulting in the output stream remaining open or a potential bug?
It appears that the close() method of HdfsOutputStream is never called unless the route/context is stopping or we are splitting the file. Do you know if this is intended? Also, do you know if the HdfsProducer is supposed keep a handle open to the hdfs file until the route/context is stopped? Cheers, Dan. -----Original Message----- From: Claus Ibsen [mailto:[email protected]] Sent: 17 October 2011 13:30 To: [email protected] Subject: Re: HDFSOutputStream not closing On Mon, Oct 17, 2011 at 2:27 PM, Daniel Baptista <[email protected]> wrote: > Hi Claus, thanks for the response. > > I can raise a ticket and will come up with a unit test. Just for completeness > the I am running windows XP and Java 1.6.0_14. > I am trying to put a file into HDFS with Hadoop version 0.20 (1 name node 2 > data nodes) where the cluster runs on GNU/Linux. > Ah thanks. So the Camel app runs on WinXP and pickup local files, which you then try to put into HDFS on a GNU/Linux box? We have seen before issues with WinXP "locking" files due to some InputStream not being closed. So this smells like that. Other OS such as Linux/OSX tend to be able to not have that "locking" issue. > Cheers, Dan. > > -----Original Message----- > From: Claus Ibsen [mailto:[email protected]] > Sent: 17 October 2011 13:06 > To: [email protected] > Subject: Re: HDFSOutputStream not closing > > On Mon, Oct 17, 2011 at 12:23 PM, Daniel Baptista > <[email protected]> wrote: >> Hi All, >> >> I have a route that is supposed to take a file from a directory and marshal >> it through to HDFS via the camel-hdfs component. Everything is working >> except the file never gets renamed from .opened (and it's empty, but I guess >> this is because the stream is open still). I am not specifying a split >> strategy and appending to files is not an issue for what I am doing as I >> always have the whole file in the Exchange. >> >> I have tried stepping through the code but I can't see where the stream is >> ever closed except when a split strategy is defined, I have tried >> MESSSAGES:0 but this results in an empty seq0 file and a seq1.opened which >> never closes (at least it is never renamed). >> >> So I guess my question is how do I just put a file into HDFS? >> >> I am using version 2.8.1 and an example endpoint URL is >> hdfs://10.1.10.159?blockSize=67108864&replication=2 >> > > Do you mind creating a JIRA ticket and if possible attach an unit test > which reproduces the issue. > Also details about your JDK, OS version used would be great. As File > Systems can vary per OS. > > >> Thanks in advance, >> >> Dan. >> > > > > -- > Claus Ibsen > ----------------- > FuseSource > Email: [email protected] > Web: http://fusesource.com > Twitter: davsclaus, fusenews > Blog: http://davsclaus.blogspot.com/ > Author of Camel in Action: http://www.manning.com/ibsen/ > -- Claus Ibsen ----------------- FuseSource Email: [email protected] Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.blogspot.com/ Author of Camel in Action: http://www.manning.com/ibsen/
