Looks like hunting down FD leak would be a good candidate for Tuesday's Hackathon :)
On Sun, Nov 27, 2011 at 9:56 AM, Todd Lipcon <[email protected]> wrote: > On Sun, Nov 27, 2011 at 9:29 AM, Ted Yu <[email protected]> wrote: > > We have many issues to tackle, build failure being one of them. > > > > Trying to reduce number of descriptors to 32k could be a goal for the > > future. For the time being, its priority is not high. > > My feeling is that this is a bug - the build failure is doing exactly > what it's supposed to: alerting us of a bug. If the builds are using > this many FDs, it's likely a leak -- just like the "out of memory" > error last week was actually do to an infinite loop spewing exceptions > in a test. > > Avoiding these errors by raising limits just hides the bug so we find > it later in production. Unless we have a good explanation that shows > that it's "just a test issue" we shouldn't ignore it or hide it by > raising ulimit. > > -Todd > > > > > On Sat, Nov 26, 2011 at 5:01 PM, lars hofhansl <[email protected]> > wrote: > > > >> I agree. The tests should not need more than 32k open files. > >> > >> > >> > >> ----- Original Message ----- > >> From: Todd Lipcon <[email protected]> > >> To: [email protected] > >> Cc: > >> Sent: Saturday, November 26, 2011 10:42 AM > >> Subject: Re: max number of open files when testing HBase on the Jenkins > >> server > >> > >> I disagree that we should raise the limit. If we're using up 32k > >> descriptors in unit tests, it's probably a real leak we should be > >> concerned about. Maybe we can add a temporary patch upstream which > >> does: > >> catch (IOException ioe) { > >> if (ioe.getMessage().contains("Too many open files")) { > >> Runtime.getRuntime().exec("lsof -u " + System.getProperty(" > user.name > >> ")); > >> } > >> } > >> or something like that? > >> > >> -Todd > >> > >> On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[email protected]> wrote: > >> > Mikhail: > >> > If you look at > >> > https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you > >> would > >> > see: > >> > > >> > max memory size (kbytes, -m) unlimited > >> > open files (-n) 32768 > >> > > >> > It is not clear why 32768 was not enough for TestAdmin. > >> > > >> > Normally we don't regard test failure resulted from 'Too many open > files' > >> > as real test failure. > >> > > >> > I agree the effective limit should be raised. > >> > > >> > Cheers > >> > > >> > On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin < > >> > [email protected]> wrote: > >> > > >> >> Hello, > >> >> > >> >> I saw this error when testing my patch on Jenkins: > >> >> > >> >> Caused by: java.io.IOException: Too many open files > >> >> at sun.nio.ch.IOUtil.initPipe(Native Method) > >> >> at > sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49) > >> >> at > >> >> > >> > sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) > >> >> at java.nio.channels.Selector.open(Selector.java:209) > >> >> at > >> >> > >> > org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42) > >> >> at > sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown > >> >> Source) > >> >> at > >> >> > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > >> >> at > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513) > >> >> at java.lang.Class.newInstance0(Class.java:355) > >> >> at java.lang.Class.newInstance(Class.java:308) > >> >> at > >> >> > org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737) > >> >> ... 55 more > >> >> > >> >> (Full log: > >> >> > >> > https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/ > >> >> ) > >> >> > >> >> I am not sure who maintains the Jenkins server, but it would be nice > to > >> >> increase the maximum number of open files for HBase unit tests. > >> >> > >> >> Thanks, > >> >> --Mikhail > >> >> > >> > > >> > >> > >> > >> -- > >> Todd Lipcon > >> Software Engineer, Cloudera > >> > >> > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
