[
https://issues.apache.org/jira/browse/HIVE-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mass Dosage updated HIVE-15965:
-------------------------------
Attachment: hive.log
Log file from a test that reproduces the issue
> Metastore incorrectly re-uses a broken database connection
> ----------------------------------------------------------
>
> Key: HIVE-15965
> URL: https://issues.apache.org/jira/browse/HIVE-15965
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: storage-2.2.0
> Reporter: Mass Dosage
> Attachments: hive.log
>
>
> *Background*
> In our setup we have a shared standalone MetaStore server running on EMR that
> is accessed by various clients (Hive CLI, HiveServer2, Spark etc.) and
> connects to an external MariaDB database for the MetaStore DB. It came to our
> attention that MetaStore (or rather the underlying DataNucleus / BoneCP
> combo) will keep re-using the same DB connections even when those get
> suddenly closed for a reason that renders them unusable.
> For instance, due to a bug in the MariaDB JDBC driver v1.3.6 (see
> https://jira.mariadb.org/browse/CONJ-270), a huge query including over 8
> thousand parameter placeholders (e.g. partition IDs in case of a
> {{get_partitions_by_expr}} function call)
> will yield a {{java.nio.BufferOverflowException}} and cause the SQL
> connection be closed by the driver itself.
> This will ultimately result in the abortion of all further MetaStore Thrift
> calls due to the failure of {{bonecp.ConnectionHandle.prepareStatement()}}.
> Such scenarios will be then caught by DataNucleus and translated to an
> appropriate {{JDOException}}, only to be "ignored" by the
> MetaStore.{{RetryingHMSHandler}} will, of course, continue retrying the
> failing operation, but this is already pointless by that time since they will
> invariably fail as long as the SQL connection remains closed. Please see the
> attached MetaStore log [^hive.log] for details
> (captured from Hive 2.1.1 running on Windows in Eclipse IDE).
> *Proposed behavior*
> We suggest that MetaStore should automatically renew the DB connection
> whenever:
> * The connection gets closed by one of the underlying frameworks
> (DataNucleus, BoneCP, JDBC driver); or
> * Query timeout is detected.
> This feature should be optional and configurable (disabled by default for
> backward compatibility). Reconnection failures could probably be treated as
> fatal errors and cause the immediate termination of MetaStore.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
