Yep, the RR pool has a similar issue. Maybe the maximum number of connection 
should be related to the number of cores on the client.
Something like 2 x #cores?


For the threadLocal case that I mentioned below it would be hard to find a 
useful hard limit, since it is hard to foresee the number of threads that will 
be accessing a region (because of the threadpool in HTable).


More generally: Should HTable have a threadpool at all? Or should the pooling 
be happening on the Connection level?


Lastly should I create an issue? :)



________________________________
From: Ted Yu <[email protected]>
To: [email protected]
Cc: lars hofhansl <[email protected]>
Sent: Friday, July 29, 2011 9:52 AM
Subject: Re: Problem with hbase.client.ipc.pool.type=threadlocal in trunk


I am waiting for Karthick to shed some light on this.


On Fri, Jul 29, 2011 at 9:47 AM, Stack <[email protected]> wrote:

Sounds good.  Issue?
>St.Ack
>
>
>On Thu, Jul 28, 2011 at 10:46 PM, Ted Yu <[email protected]> wrote:
>> For HBaseClient, at least the javadoc doesn't match:
>>
>>   * @param config configuration
>>   * @return either a {@link PoolType#Reusable} or {@link
>> PoolType#ThreadLocal}
>>   */
>>  private static PoolType getPoolType(Configuration config) {
>>    return
>> PoolType.valueOf(config.get(HConstants.HBASE_CLIENT_IPC_POOL_TYPE),
>>        PoolType.RoundRobin, PoolType.ThreadLocal);
>>
>> I think for RoundRobinPool, we shouldn't allow maxSize to be
>> Integer#MAX_VALUE. Otherwise connection explosion described by Lars may
>> incur.
>>
>> Cheers
>>
>> On Thu, Jul 28, 2011 at 10:27 PM, lars hofhansl <[email protected]> wrote:
>>
>>> Yeah I looked at that one too, but I assumed that the rationale was to keep
>>> creating connectionsuntil the pool is full, which makes
>>> some sense for a round robin pool. Otherwise it is not clear to me how the
>>> pool grow at all?
>>>
>>>
>>> (And at least it can be usefully controlled with the pool.size parameter.)
>>>
>>>
>>> Cheers indeed
>>>
>>>
>>> ----- Original Message -----
>>> From: Ted Yu <[email protected]>
>>> To: [email protected]
>>> Cc:
>>> Sent: Thursday, July 28, 2011 10:07 PM
>>> Subject: Re: Problem with hbase.client.ipc.pool.type=threadlocal in trunk
>>>
>>> Looking at PoolMap, I have some question as well.
>>> For RoundRobinPool, I don't understand the following method:
>>>     public R get() {
>>>       if (size() < maxSize) {
>>>         return null;
>>>       }
>>> Should the above condition be?
>>>       if (size() <= 0) {
>>>
>>> For the issue Lars raised, I prefer solution number 2.
>>>
>>> Cheers
>>>
>>> On Thu, Jul 28, 2011 at 9:12 PM, lars <[email protected]> wrote:
>>>
>>> > Looking at HBaseClient.getConnection(...) I see this:
>>> > ...
>>> >      synchronized (connections) {
>>> >        connection = connections.get(remoteId);
>>> >        if (connection == null) {
>>> >          connection = new Connection(remoteId);
>>> >          connections.put(remoteId, connection);
>>> >        }
>>> >      }
>>> > ...
>>> >
>>> > At the same time PoolMap.ThreadLocalPool.put is defined like this:
>>> >    public R put(R resource) {
>>> >      R previousResource = get();
>>> >      if (previousResource == null) {
>>> > ...
>>> >        if (poolSize.intValue() >= maxSize) {
>>> >          return null;
>>> >        }
>>> > ...
>>> >    }
>>> >
>>> > So... If the ThreadLocalPool reaches its capacity it always returns null
>>> > and hence all new threads will create a
>>> > new connection every time getConnection is called!
>>> >
>>> > I have also verified with a test program that works fine as long as the
>>> > number of client threads (which include
>>> > the threads in HTable's threadpool of course) is < poolsize. Once that is
>>> > no longer the case the number of
>>> > connections "explodes" and the program dies with OOMEs (mostly because
>>> each
>>> > Connection is associated with
>>> > yet another thread).
>>> >
>>> > It's not clear what should happen, though. Maybe (1) the ThreadLocalPool
>>> > should not have a limit, or maybe
>>> > (2) allocations past the pool size should throw an exception (i.e.
>>> there's
>>> > a hard limit), or maybe (3) in that case
>>> > a single connection is returned for all threads while the pool it over
>>> its
>>> > limit or (4) we start round robin with the other
>>> > connection in the other thread locals.
>>> >
>>> > For #1 means that the number of client threads needs to be more carefully
>>> > managed by the client app.
>>> > In this case it would also be somewhat pointless that Connection have
>>> their
>>> > own threads, we just pass stuff
>>> > between threads.
>>> > #2 would work, but puts more logic in the client.
>>> > #3 would lead to hard to debug performance issues.
>>> > And #4 is messy :)
>>> >
>>> > Any other options?
>>> >
>>> > Maybe for now just do #2 (or #1)?
>>> >
>>> > -- Lars
>>> >
>>> >
>>>
>>>
>>
>

Reply via email to