On Mon, Oct 17, 2011 at 2:48 PM, Daniel Baptista <[email protected]> wrote: > I put a breakpoint and called close() on the HdfsOutputStream (via the > display window in eclipse) and then file was created (correctly named and > with the expected contents) so it appears that the issue is with either a > non-terminating process resulting in the output stream remaining open or a > potential bug? > > It appears that the close() method of HdfsOutputStream is never called unless > the route/context is stopping or we are splitting the file. Do you know if > this is intended? > > Also, do you know if the HdfsProducer is supposed keep a handle open to the > hdfs file until the route/context is stopped?
No I dont think its intended. Seems like the producer should pickup the file, process it, and then close any file handle on it. I suspect there is a bug. Could you raise a JIRA? Its just that those "no close on File handle" issues is more apparent on WinXP, than on Linux or OSX, and hence not discovered as fast. > > Cheers, Dan. > > -----Original Message----- > From: Claus Ibsen [mailto:[email protected]] > Sent: 17 October 2011 13:30 > To: [email protected] > Subject: Re: HDFSOutputStream not closing > > On Mon, Oct 17, 2011 at 2:27 PM, Daniel Baptista > <[email protected]> wrote: >> Hi Claus, thanks for the response. >> >> I can raise a ticket and will come up with a unit test. Just for >> completeness the I am running windows XP and Java 1.6.0_14. >> I am trying to put a file into HDFS with Hadoop version 0.20 (1 name node 2 >> data nodes) where the cluster runs on GNU/Linux. >> > > Ah thanks. So the Camel app runs on WinXP and pickup local files, > which you then try to put into HDFS on a GNU/Linux box? > > We have seen before issues with WinXP "locking" files due to some > InputStream not being closed. So this smells like that. > Other OS such as Linux/OSX tend to be able to not have that "locking" issue. > > > >> Cheers, Dan. >> >> -----Original Message----- >> From: Claus Ibsen [mailto:[email protected]] >> Sent: 17 October 2011 13:06 >> To: [email protected] >> Subject: Re: HDFSOutputStream not closing >> >> On Mon, Oct 17, 2011 at 12:23 PM, Daniel Baptista >> <[email protected]> wrote: >>> Hi All, >>> >>> I have a route that is supposed to take a file from a directory and marshal >>> it through to HDFS via the camel-hdfs component. Everything is working >>> except the file never gets renamed from .opened (and it's empty, but I >>> guess this is because the stream is open still). I am not specifying a >>> split strategy and appending to files is not an issue for what I am doing >>> as I always have the whole file in the Exchange. >>> >>> I have tried stepping through the code but I can't see where the stream is >>> ever closed except when a split strategy is defined, I have tried >>> MESSSAGES:0 but this results in an empty seq0 file and a seq1.opened which >>> never closes (at least it is never renamed). >>> >>> So I guess my question is how do I just put a file into HDFS? >>> >>> I am using version 2.8.1 and an example endpoint URL is >>> hdfs://10.1.10.159?blockSize=67108864&replication=2 >>> >> >> Do you mind creating a JIRA ticket and if possible attach an unit test >> which reproduces the issue. >> Also details about your JDK, OS version used would be great. As File >> Systems can vary per OS. >> >> >>> Thanks in advance, >>> >>> Dan. >>> >> >> >> >> -- >> Claus Ibsen >> ----------------- >> FuseSource >> Email: [email protected] >> Web: http://fusesource.com >> Twitter: davsclaus, fusenews >> Blog: http://davsclaus.blogspot.com/ >> Author of Camel in Action: http://www.manning.com/ibsen/ >> > > > > -- > Claus Ibsen > ----------------- > FuseSource > Email: [email protected] > Web: http://fusesource.com > Twitter: davsclaus, fusenews > Blog: http://davsclaus.blogspot.com/ > Author of Camel in Action: http://www.manning.com/ibsen/ > -- Claus Ibsen ----------------- FuseSource Email: [email protected] Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.blogspot.com/ Author of Camel in Action: http://www.manning.com/ibsen/
