[
https://issues.apache.org/jira/browse/SENTRY-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181744#comment-16181744
]
Alexander Kolbasov commented on SENTRY-1965:
--------------------------------------------
[~_mcarter_] [~arjunmishra13] [~spena] [~btowles] FYI
> sentry server database database Destroy connection exception (Galera cluster)
> -----------------------------------------------------------------------------
>
> Key: SENTRY-1965
> URL: https://issues.apache.org/jira/browse/SENTRY-1965
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
> Affects Versions: 1.5.1
> Environment: Sentry 1.5.1
> CDH 5.9.1
> Mysql 5.6.35
> Galera 25.18
> Reporter: Steven Hancz
> Labels: galera
>
> We have implemented an HA solution for the Sentry server database.
> Basically instead of using a single MySQL server we have a Galera cluster
> that is accessed via a DNS load balanced VIP. So that if one MySQL server
> stops working the VIP will detect and send the DB request to the surviving
> node. A similar set up is working for the HIVE metastore.
> However we noticed that Sentry just like spark uses the BoneCP connection
> pool to connect to the database. There are some hard codded configuration
> options in the bonecp-default-config.xml that are causing issues with Sentry.
> idleConnectionTestPeriodInMinutes default 240 minutes
> idleMaxAgeInMinutes default 60 minutes
> Based on this BonceCP will test each idle connection every 240 minutes
> but an idle connection is closed after 60 minutes (second parameter) so the
> connection testing will never take place as the connection will be closed
> after 60 minutes. The test takes place every 240 minutes.
> However in an HA configuration with a VIP you can set the connection time out
> and how often to test for target availability.
> We had the exact same problem for hive there the work around was to include a
> second configuration file for BoneCP called bonecp-config.xml. This was added
> to the hive server jar. The second config file (bonecp-config.xml) contains
> idleConnectionTestPeriodInMinutes 1
> idleMaxAgeInMinutes 5
> So that every connection is tested every minute and an idle connection is
> closed after 5 minutes. But since we test it every minute they will be kept
> alive.
> So the question is how to enable a similar setting for Sentry ?
> With default boneCP configuration and Galera cluster in the back end Sentry
> is returning the following error:
> Sep 26, 7:30:39.900 AM ERROR
> com.jolbox.bonecp.ConnectionTesterThread
> Destroy connection exception
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution unknown.
> at sun.reflect.GeneratedConstructorAccessor41.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
> at com.mysql.jdbc.Util.getInstance(Util.java:387)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
> at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4634)
> at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4263)
> at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1519)
> at
> com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:396)
> at
> com.jolbox.bonecp.ConnectionTesterThread.closeConnection(ConnectionTesterThread.java:155)
> at
> com.jolbox.bonecp.ConnectionTesterThread.run(ConnectionTesterThread.java:95)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> From our research it appears that sentry server is using BoneCP class but in
> more than one location. Changing the parameters in BoneCP for Sentry alone is
> does not appear to be sufficient. Trace file shows that parameters are not
> changed and time outs are default boncecp parameter. Where else do we have to
> change boneCP config?
> Regards,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
