Let me try . Thanks very much for your help. :D Thanks & Best regards LiuJinglong
在 2010年7月30日 下午3:35，Angus He <[email protected]>写道： > Close the scannner will not close the store file. > > Probably you could increase the memstore size and region file size. > And hbase.hstore.compactionThreshold and > hbase.hstore.blockingStoreFiles might help too. > > 2010/7/30 baggio liu <[email protected]>: > > In our cluster, there're thousands of region and each region has 5-8 > store > > files, so the connection number is very terrible. > > And if we close the store file after scanning , the connection number may > > bring down some region servers. (In my cluster , my region server went > down > > caused by socket OOM, our fd limit set to 30000, and tcp_mem set to 2G). > > We want to reuse these connections in DFSClient, do you have some more > > graceful solutions ? > > > > > > Thanks & Best regards > > LiuJinglong > > > > > > 在 2010年7月30日 下午2:51，Angus He <[email protected]>写道： > > > >> Hi Baggio, > >> > >> > >> > 2. During scanning, it'll new StoreFile one by one. And in > constructor > >> of > >> > StoreFile , HFile.Reader will be created. HFile.Reader act as > DFSClient, > >> > it'll keep a connection with DataNode when something should be read. > >> > >> Yes, There is a HFile.Reader opend in StoreFile constructor. > >> But I am afraid there is only one instance in HBase for each > >> StoreFile. And I just confirmed this > >> in my dev machine. (HBase running in standalone mode). After many scan > >> operations, still only one > >> instance for each StoreFile. > >> > >> > 1.After checking code, and I've seen in scanner close method, > >> > HFile.Reader has not been closed. Is that used for reuse HFile ? > >> > I try to close the scanner explictly, but it cause meta region > can > >> not > >> > be loaded when start up. Both get() and scan() use the same > scanner... > >> > >> It seems there is only one HFile.Reader for each StoreFile, and the > >> HFile.Reader instance is shared by multiple scannners. > >> > >> As for closing the scannner explicitly, it works all right in our > >> case. And really do not know what happened in your case. > >> > >> By the way, how do you get the scanner,by HTable or other ways? > >> > >> > >> > > >> > Thanks & Best regards > >> > LiuJinglong > >> > > >> > 在 2010年7月30日 上午11:33，Angus He <[email protected]>写道： > >> > > >> >> 1. try to close the scanner explictly? > >> >> > >> >> 2. I do not think HBase will issue a new connection for each > StoreFile > >> >> for the scan operation. > >> >> > >> >> 2010/7/29 baggio liu <[email protected]>: > >> >> > Hi all， > >> >> > We have 53 machines in our hbase cluster and run 6 clients to > scan > >> a > >> >> > table. During scanning, we found when a region is scanning , it'll > new > >> >> > StoreFile object, create a connection to datanode (in fact , create > >> HFile > >> >> > Reader), so the number of connection increases by the number of > store > >> >> files. > >> >> > We have many store files ,(cross several regions and has not reach > >> minor > >> >> > compaction thredshould), too many connections has been created. And > >> after > >> >> > scanning, the connection will not closed. > >> >> > As the result of it , a machine which act as region server has > too > >> >> high > >> >> > system CPU, and hung for a long time. > >> >> > My question is : > >> >> > 1. Why we don't close connection ( in fact, we don't close > >> >> > HFile.Reader)after we complete to scan table ? We wanna to reuse > >> >> connection > >> >> > in the next scan ?? > >> >> > 2. How can we limit the connection number ? > >> >> > > >> >> > Thanks & Best regards > >> >> > LiuJinglong > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Regards > >> >> Angus > >> >> > >> > > >> > >> > >> > >> -- > >> Regards > >> Angus > >> > > > > > > -- > Regards > Angus >
