The solution is to use a Singleton to hold the pool. Guard the initialization of Singleton such that when the 'N' tasks on a worker try to initialize it, only one of them succeeds and the rest of them see that the pool is already initialized. This you would do once in the prepare() method. There is contention but only once during worker initialization.
On Sat, Sep 6, 2014 at 4:06 AM, Chen Wang <[email protected]> wrote: > I think its a common problem in distributed system: > I only need a redis pool in one VM(or less complicated, one redis pool for > a bolt class), and I want to be able to change the host/port in the config > file without recompiling code. > > so when I submit the topology, i read from file, and put into storm > config, so the config (host/port) are all available in prepare() of bolt. > Normally, one would think of doing the following: > class MyBolt{ > static JedisPool jedispool; > > public void prepare(Map stormConf, TopologyContext context, > > OutputCollector collector) { > jedisPool = new JedisPool(poolConfig, > > stormConfig.get(host), > > stormConfig.get(port)); > } > > But if MyBolt's parallelism is 200, then I will still create 200 > JedisPools, right? and I would not know which one is actually used during > execute().. > > So how do you guys normally handle this scenario? It would be the same > problem for creating db connection pool etc. > > Thanks for your input. > Chen > > >
