[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507345#comment-13507345
]
Bryan Baugher commented on HBASE-6956:
--
We have also ran into this issue specifically during major failure of the
cluster (i.e. hmaster, zookeeper, regionservers are down).
[~amuraru] did you manage to find a good way to flush a bad table from your
pool when the connection becomes closed?
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507353#comment-13507353
]
Igor Yurinok commented on HBASE-6956:
-
We implemented our own HTableFactory and HTable which able to check whether
connection closed:
{code}
public class HumbleHTableFactory implements HTableInterfaceFactory {
@Override
public HTableInterface createHTableInterface(Configuration config, byte[]
tableName) {
try {
return new HumbleHTable(config, tableName);
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
@Override
public void releaseHTableInterface(HTableInterface table) {
try {
table.close();
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
}
}
{code}
{code}
public class HumbleHTable extends HTable {
private static final Log log = LogFactory.getLog(HumbleHTable.class);
public HumbleHTable(String tableName) throws IOException {
super(tableName);
}
public HumbleHTable(byte[] tableName) throws IOException {
super(tableName);
}
public HumbleHTable(Configuration conf, String tableName) throws
IOException {
super(conf, tableName);
}
public HumbleHTable(Configuration conf, byte[] tableName) throws
IOException {
super(conf, tableName);
}
/**
* Harmless clean-up - HConnection isn't touched. Only the executor pool is
shut down
*
* @throws IOException
*/
@Override
public void close() throws IOException {
if (isClosed()) {
return;
}
flushCommits();
ExecutorService pool = getPool();
if (pool != null) {
pool.shutdown();
}
setClosed();
}
private boolean isClosed() {
try {
return getSuperField(closed).getBoolean(this);
} catch (Exception e) {
log.error(e);
return false;
}
}
private void setClosed() {
try {
getSuperField(closed).setBoolean(this, true);
} catch (Exception e) {
log.error(e);
}
}
private ExecutorService getPool() {
try {
return (ExecutorService) getSuperField(pool).get(this);
} catch (Exception e) {
log.error(e);
return null;
}
}
/**
* Loads stuff from the parent class via reflection
*/
private Field getSuperField(String fieldName) throws NoSuchFieldException {
Field field = HTable.class.getDeclaredField(fieldName);
field.setAccessible(true);
return field;
}
}
{code}
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507413#comment-13507413
]
stack commented on HBASE-6956:
--
This seems like an ugly enough oversight on our part. Any chance of a patch to
address it? Could we change HConnection Interface to make it easier getting a
new one [~amuraru]? (I like the name of your class [~iyurinok])
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507634#comment-13507634
]
Micah Whitacre commented on HBASE-6956:
---
Igor, can you explain how you are detecting the connection is closed? Your
isClosed() method only checks to see if the HTable boolean value has been
closed which is independent of the HConnection.
Additionally why do you avoid trying to close the HConnection by informing the
HConnectionManager that the connection is stale?
The last question is how does this solve the problem of evicting
HTableInterface instances from the pool that are no longer valid? Altering the
HTableInterfaceFactory + HTableInterface implementation only solves the problem
of new instances needed to be created.
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507639#comment-13507639
]
Adrian Muraru commented on HBASE-6956:
--
There is an ongoing discussion on HBASE-6580 addressing this issue, a broader
scope but would resolve the broken HConnection discussed here.
The agreement there was to deprecate HTablePool and rely on lightweight (read
cheap) HTable (HBASE-4805) retrieved from a new HConnection#getTable()
I have a patch on HBASE-6580 that proposes a new HTableInterfaceFactory
:https://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch
and can be used as a custom factory when creating the HTablePool
{noformat}org.apache.hadoop.hbase.client.HTablePool.HTablePool(Configuration
config, int maxSize, HTableInterfaceFactory tableFactory){noformat}
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491365#comment-13491365
]
shengjie min commented on HBASE-6956:
-
[~amuraru]what version you are using?
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491367#comment-13491367
]
Adrian Muraru commented on HBASE-6956:
--
[~shengjie_min] 0.92.2
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13491169#comment-13491169
]
Adrian Muraru commented on HBASE-6956:
--
I'm seeing a similar stack trace when using HTablePool:
{code}
java.lang.RuntimeException: java.io.IOException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5c429be9
closed
at
org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
~[hbase-0.92.2-26.jar:0.92.2-26]
at
org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
~[hbase-0.92.2-26.jar:0.92.2-26]
at
org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
~[hbase-0.92.2-26.jar:0.92.2-26]
at
org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
~[hbase-0.92.2-26.jar:0.92.2-26]
{code}
The issue is not with a closed HTable instance returned back to the pool.
Instead, following my investigation it is caused by the shared *HConnection*
instance being closed.
Once you get into this state - the HTablePool#getTable() will always fail and
never recover.
A possible fix is to drop the HConnection used by HTablePool#getTable() in case
of an IOException.
Not trivial though as it is created based on a private Configuration instance
in the pool :)
{code}
this.connection = HConnectionManager.getConnection(conf)
{code}
A more brutal way of escaping this state is to drop the pool altogether a
create a new one (not really doable in all cases depending on the usage)
I'll try to come up with a patch to clean up the faulty HConnection and keep
the pool up.
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487006#comment-13487006
]
shengjie min commented on HBASE-6956:
-
I still have this issue even in 0.92.1. any updates?
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487013#comment-13487013
]
Ted Yu commented on HBASE-6956:
---
For putTable(), I see this javadoc:
{code}
/**
* This method is not needed anymore, clients should call
* HTableInterface.close() rather than returning the tables to the pool
*
* @param table
* the proxy table user got from pool
* @deprecated
*/
public void putTable(HTableInterface table) throws IOException {
{code}
@Igor, @Shengjie:
Can you tell us whether you followed the above suggestion ?
Thanks
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13487037#comment-13487037
]
shengjie min commented on HBASE-6956:
-
In my case I am using HTablePool, my understanding is the following:
so when the hbase query comes,
if pool is avaiable:
I get a PooledHtable instance, once I am done with query, I call
HtablePool.PooledHtable.close(), it will return PooledHtable instance to the
pool.
if pool is out:
I get a normal Htable instance, once I am done with query, I call
Htable.close(), it will just release Htable instance.
I have no clue why I ended up having closed HConnectionImplementation in the
pool.
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6956) Do not return back to HTablePool closed connections
[
https://issues.apache.org/jira/browse/HBASE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472779#comment-13472779
]
stack commented on HBASE-6956:
--
You have a patch Igor?
Do not return back to HTablePool closed connections
---
Key: HBASE-6956
URL: https://issues.apache.org/jira/browse/HBASE-6956
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.90.6
Reporter: Igor Yurinok
Sometimes we see a lot of Exception about closed connections:
{code}
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
org.apache.hadoop.hbase.client.ClosedConnectionException:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@553fd068
closed
{code}
After investigation we assumed that it occurs because closed connection
returns back into HTablePool.
For our opinion best solution is check whether the table is closed in method
HTablePool.putTable and if true don't add it into the queue and release such
HTableInterface.
But unfortunatly right now there are no access to HTable#closed field through
HTableInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
