Hi. In my testings, I typically opened between 20 and 40 concurrent streams.
Regards. 2009/6/23 Raghu Angadi <rang...@yahoo-inc.com> > Stas Oskin wrote: > >> Hi. >> >> Any idea if calling System.gc() periodically will help reducing the amount >> of pipes / epolls? >> > > since you have HADOOP-4346, you should not have excessive epoll/pipe fds > open. First of all do you still have the problem? If yes, how many hadoop > streams do you have at a time? > > System.gc() won't help if you have HADOOP-4346. > > Ragu. > > > Thanks for your opinion! >> >> 2009/6/22 Stas Oskin <stas.os...@gmail.com> >> >> Ok, seems this issue is already patched in the Hadoop distro I'm using >>> (Cloudera). >>> >>> Any idea if I still should call GC manually/periodically to clean out all >>> the stale pipes / epolls? >>> >>> 2009/6/22 Steve Loughran <ste...@apache.org> >>> >>> Stas Oskin wrote: >>>> >>>> Hi. >>>> >>>>> So what would be the recommended approach to pre-0.20.x series? >>>>> >>>>> To insure each file is used only by one thread, and then it safe to >>>>> close >>>>> the handle in that thread? >>>>> >>>>> Regards. >>>>> >>>>> good question -I'm not sure. For anythiong you get with >>>> FileSystem.get(), >>>> its now dangerous to close, so try just setting the reference to null >>>> and >>>> hoping that GC will do the finalize() when needed >>>> >>>> >> >
