On Fri, Jul 3, 2009 at 11:20 AM, David B. Ritch <[email protected]>wrote:
> I have been told that it is not a good idea to keep HDFS files open for > a long time. The reason sounded like a memory leak in the name node - > that over time, the resources absorbed by an open file will increase. > > Is this still an issue with Hadoop-0,19.x and 0-20.x? Was it ever an > issue? > You probably need the patch in HDFS-127 if you want to keep files open over a long period of time. As is, any 3 hiccups hobble an open DFSClient stream. This patch, roughly, resets counters on success. Its a patch for DFSClient so you can get away with applying it to the hadoop your client uses if you don't -- or can't -- apply it cross cluster (We recommend running with it in our application which keeps long-running open streams on hdfs files). St.Ack
