Shankar, seems that those close_wait are bound to ipv6, have you tried to disable ipv6 first? (-Djava.net.preferIPv4Stack=true)
Regards, Esteban. -- Cloudera, Inc. On Thu, Jul 24, 2014 at 5:02 AM, Shankar hiremath < [email protected]> wrote: > I am using Hbase 0.98.3 and Hadoop 2.4.1 > > Regards > -Shankar > > > This e-mail and its attachments contain confidential information from > HUAWEI, which is intended only for the person or entity whose address is > listed above. Any use of the information contained herein in any way > (including, but not limited to, total or partial disclosure, reproduction, > or dissemination) by persons other than the intended recipient(s) is > prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it! > > > > > > -----Original Message----- > From: Qiang Tian [mailto:[email protected]] > Sent: 24 July 2014 PM 03:19 > To: [email protected] > Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not > getting cleared > > what is your hbase version? > the close_wait is in regionserver process, probably rpc connection close() > is not call due to some reason? can you pastebin the related regionserver > log? > > > On Thu, Jul 24, 2014 at 3:16 PM, ramkrishna vasudevan < > [email protected]> wrote: > > > Sorry about that. > > https://issues.apache.org/jira/browse/HBASE-9393. Similar issue has > > been reported. > > > > What version of HBase and HDFS are you using? See if things are similar. > > > > Regards > > Ram > > > > > > On Thu, Jul 24, 2014 at 12:33 PM, Shankar hiremath < > > [email protected]> wrote: > > > > > Hi Ram, > > > > > > Actually It is not file FD, it is socket FD which is in CLOSE_WAIT > > > state (tcp socket connection) > > > > > > > java 18028 shankar1 194u IPv6 18548560 0t0 TCP > > > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > > > Regards > > > -Shankar > > > > > > > > > This e-mail and its attachments contain confidential information > > > from HUAWEI, which is intended only for the person or entity whose > > > address is listed above. Any use of the information contained herein > > > in any way (including, but not limited to, total or partial > > > disclosure, > > reproduction, > > > or dissemination) by persons other than the intended recipient(s) is > > > prohibited. If you receive this e-mail in error, please notify the > > > sender by phone or email immediately and delete it! > > > > > > > > > > > > > > > -----Original Message----- > > > From: ramkrishna vasudevan [mailto:[email protected]] > > > Sent: 24 July 2014 PM 12:23 > > > To: [email protected] > > > Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not > > > getting cleared > > > > > > Does your regions have store files in them? > > > In HBase all the store files are opened and the readers are kept > > > open always when the region opening happens. When there is a > > > compaction happening the count of those file handlers open gets > > > reduced based on the number of files compacted. > > > > > > Regards > > > Ram > > > > > > > > > On Thu, Jul 24, 2014 at 12:19 PM, Shankar hiremath < > > > [email protected]> wrote: > > > > > > > Dear All, > > > > > > > > Observation: > > > > I have HBase cluster with Kerberos enabled, when the Region Server > > > > startups then we observed some 20-30 socket fds are in CLOSE_WAIT > > > > state, After that when the Region Server starts opening then the > > > > socket fds in CLOSE_WAIT starts increasing gradually (almost same > > > > as number of regions opened by the region server) And all these > > > > CLOSE_WAIT states are not getting cleared up, > > > > > > > > /hbase> lsof -i | grep `jps | grep RegionServer | cut -d " " -f1` > > > > java 18028 shankar1 118u IPv6 18552894 0t0 TCP > > > > XX-XX-XX-XX:60020 (LISTEN) > > > > java 18028 shankar1 160u IPv6 18548520 0t0 TCP *:60030 > > > (LISTEN) > > > > java 18028 shankar1 167u IPv6 18548522 0t0 TCP > > > > XX-XX-XX-XX:42534-> XX-XX-XX-XX:eforward (ESTABLISHED) > > > > java 18028 shankar1 172u IPv6 18552916 0t0 TCP > > > > XX-XX-XX-XX:42535-> XX-XX-XX-XX:eforward (ESTABLISHED) > > > > java 18028 shankar1 173u IPv6 18551227 0t0 TCP > > > > XX-XX-XX-XX:49646->XX-XX-XX-XX:60000 (ESTABLISHED) > > > > java 18028 shankar1 178u IPv6 18551237 0t0 TCP > > > > XX-XX-XX-XX:62668->XX-XX-XX-XX:busboy (ESTABLISHED) > > > > java 18028 shankar1 185u IPv6 18548549 0t0 TCP > > > > XX-XX-XX-XX:21856-> XX-XX-XX-XX:eforward (ESTABLISHED) > > > > java 18028 shankar1 187u IPv6 18548558 0t0 TCP > > > > XX-XX-XX-XX:62673->XX-XX-XX-XX:busboy (ESTABLISHED) > > > > java 18028 shankar1 188u IPv6 18601323 0t0 TCP > > > > XX-XX-XX-XX:63168->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 189u IPv6 18601322 0t0 TCP > > > > XX-XX-XX-XX:63167->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 190u IPv6 18601324 0t0 TCP > > > > XX-XX-XX-XX:63169->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 191r IPv6 18592423 0t0 TCP > > > > XX-XX-XX-XX:63087->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 193u IPv6 18593210 0t0 TCP > > > > XX-XX-XX-XX:63090->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 194u IPv6 18548560 0t0 TCP > > > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 195u IPv6 18592428 0t0 TCP > > > > XX-XX-XX-XX:63093->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 196u IPv6 18593218 0t0 TCP > > > > XX-XX-XX-XX:63096->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 197u IPv6 18591423 0t0 TCP > > > > XX-XX-XX-XX:63105->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 201u IPv6 18592431 0t0 TCP > > > > XX-XX-XX-XX:63099->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 202u IPv6 18592433 0t0 TCP > > > > XX-XX-XX-XX:63102->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > java 18028 shankar1 203u IPv6 18552317 0t0 TCP > > > > XX-XX-XX-XX:62681->XX-XX-XX-XX:busboy (CLOSE_WAIT) > > > > > > > > > > > > > > ............................................................................................................ > > > > > > > > > > > > > > ............................................................................................................ > > > > > > > > > > > > > > ............................................................................................................ > > > > > > > > > > > > > > ............................................................................................................ > > > > > > > > > > > > > > ............................................................................................................ > > > > > > > > any input or suggestion will be helpful, or is it a bug. > > > > > > > > Regards > > > > -Shankar > > > > > > > > > > > > [X] > > > > This e-mail and its attachments contain confidential information > > > > from HUAWEI, which is intended only for the person or entity whose > > > > address is listed above. Any use of the information contained > > > > herein in any way (including, but not limited to, total or partial > > > > disclosure, reproduction, or dissemination) by persons other than > > > > the intended > > > > recipient(s) is prohibited. If you receive this e-mail in error, > > > > please notify the sender by phone or email immediately and delete it! > > > > [X] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
