Storm 0.9.1 crashing possibly too many connections to hbase 0.96 - Red Hat 6.4

[Adam Pritchard]

Hi everyone,


I have been having issue with Storm running for last month where one of my
topologies crashes without giving any warning or logging any kind of error
message.


The topology has a bolt where it saves data to hbase.  In another bolt I am
saving data to Solr.


I believe that the bolt with hbase is causing the issue, perhaps because I
am not closing connections correctly, or making use of hbase connections in
the right way.


I have another topology handling the same amount of data, but making no
connections to any db's and it runs with no problems, never crashes.


How can I improve performance of my Hbase connections?



Storm version 0.9.1-incubating

Hbase version: HBase 0.96.1.2.0.11.0-1-hadoop2

OS: Red Hat Enterprise Linux Server release 6.4 (Santiago)


Num workers 2

Num executors 12

Num tasks 12



I have included my prepare and cleanup methods.  I just read that cleanup
is not guaranteed to run, if that is the case should I close the
connections within execute every time?


Thank you for any information you can provide.

- Adam


*public void *prepare(Map stormConf, TopologyContext context,
OutputCollector collector)
{
*indexerConfig *=
ConfigurationManager.*convertJsonSerializedMapToConfig*(((Map<Object,
Object>) stormConf.get(Constants.*INDEXER_CONFIG*)));
Configuration conf = *new *Configuration();
conf.set(*"hbase.zookeeper.quorum"*, StringUtils.*join*(*indexerConfig*
.getStringArray(Constants.*HBASE_ZOOKEEPERS*), *','*));
conf.set(*"hbase.zookeeper.property.clientPort"*, *indexerConfig*
.getString(Constants.*HBASE_ZOOKEEPER_PORT*));
*tablePool *= *new *HTablePool(conf, Integer.*parseInt*(*indexerConfig*
.getString(Constants.*HBASE_THREAD_SIZE*)));
*tsdbUidTable *= *tablePool*.getTable(*indexerConfig*.getString(Constants.
*HBASE_UID_TABLE*));
*tsdbUidTable*.setAutoFlushTo(*true*);
*keyHandler *= *new *TsdbKeyHandler(*tsdbUidTable*);
*tsdbDataTable *= *tablePool*.getTable(*indexerConfig*.getString(Constants.
*HBASE_METRIC_TABLE*));
*tsdbDataTable*.setAutoFlushTo(*true*);
*this*.*outputCollector *= collector;
}




public void cleanup()
{
   try
   {
      tsdbUidTable.close();
   }
   catch (IOException ioe)
   {
      logger.error("IOException occurred in cleanup() method of
DataIndexerBolt class while closing UID table.", ioe);
   }
   try
   {
      tsdbDataTable.close();
   }
   catch (IOException ioe)
   {
      logger.error("IOException occurred in cleanup() method of
DataIndexerBolt class while closing data table.", ioe);
   }
   try
   {
      tablePool.close();
   }
   catch (IOException ioe)
   {
      logger.error("IOException occurred in cleanup() method of
DataIndexerBolt class while closing hbase table pool.", ioe);
   }
}