[
https://issues.apache.org/jira/browse/HIVE-25522?focusedWorklogId=669858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-669858
]
ASF GitHub Bot logged work on HIVE-25522:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Oct/21 00:34
Start Date: 26/Oct/21 00:34
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #2647:
URL: https://github.com/apache/hive/pull/2647#discussion_r736058006
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -5567,10 +5570,16 @@ private void removeTxnsFromMinHistoryLevel(Connection
dbConn, List<Long> txnids)
}
}
- private static synchronized DataSource setupJdbcConnectionPool(Configuration
conf, int maxPoolSize, long getConnectionTimeoutMs) throws SQLException {
+ private static DataSource setupJdbcConnectionPool(Configuration conf, int
maxPoolSize, long getConnectionTimeoutMs) {
Review comment:
hmm why remove `synchronized`?
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -5567,10 +5570,16 @@ private void removeTxnsFromMinHistoryLevel(Connection
dbConn, List<Long> txnids)
}
}
- private static synchronized DataSource setupJdbcConnectionPool(Configuration
conf, int maxPoolSize, long getConnectionTimeoutMs) throws SQLException {
+ private static DataSource setupJdbcConnectionPool(Configuration conf, int
maxPoolSize, long getConnectionTimeoutMs) {
DataSourceProvider dsp =
DataSourceProviderFactory.tryGetDataSourceProviderOrNull(conf);
if (dsp != null) {
- return dsp.create(conf);
+ try {
+ return dsp.create(conf);
+ } catch (SQLException e) {
+ String msg = "Unable to instantiate JDBC connection pooling, " +
e.getMessage();
Review comment:
maybe `LOG.error("Unable to instantiate JDBC connection pooling", e);`?
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##########
@@ -368,33 +368,37 @@ public TxnHandler() {
public void setConf(Configuration conf){
this.conf = conf;
+ int maxPoolSize = MetastoreConf.getIntVar(conf,
ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+ long getConnectionTimeoutMs = 30000;
synchronized (TxnHandler.class) {
if (connPool == null) {
- Connection dbConn = null;
- // Set up the JDBC connection pool
- try {
- int maxPoolSize = MetastoreConf.getIntVar(conf,
ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
- long getConnectionTimeoutMs = 30000;
- connPool = setupJdbcConnectionPool(conf, maxPoolSize,
getConnectionTimeoutMs);
- /*the mutex pools should ideally be somewhat larger since some
operations require 1
+ connPool = setupJdbcConnectionPool(conf, maxPoolSize,
getConnectionTimeoutMs);
+ }
+
+ if (connPoolMutex == null) {
+ /*the mutex pools should ideally be somewhat larger since some
operations require 1
connection from each pool and we want to avoid taking a connection
from primary pool
and then blocking because mutex pool is empty. There is only 1
thread in any HMS trying
to mutex on each MUTEX_KEY except MUTEX_KEY.CheckLock. The
CheckLock operation gets a
connection from connPool first, then connPoolMutex. All others, go
in the opposite
order (not very elegant...). So number of connection requests for
connPoolMutex cannot
exceed (size of connPool + MUTEX_KEY.values().length - 1).*/
- connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize +
MUTEX_KEY.values().length, getConnectionTimeoutMs);
- dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+ connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize +
MUTEX_KEY.values().length, getConnectionTimeoutMs);
+ }
+
+ if (dbProduct == null) {
+ try (Connection dbConn =
getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
determineDatabaseProduct(dbConn);
- sqlGenerator = new SQLGenerator(dbProduct, conf);
} catch (SQLException e) {
- String msg = "Unable to instantiate JDBC connection pooling, " +
e.getMessage();
+ String msg = "Unable to determine database product, " +
e.getMessage();
Review comment:
ditto: `LOG.error("Unable to determine database product", e);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 669858)
Time Spent: 9h 10m (was: 9h)
> NullPointerException in TxnHandler
> ----------------------------------
>
> Key: HIVE-25522
> URL: https://issues.apache.org/jira/browse/HIVE-25522
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Affects Versions: 3.1.2
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Environment: Using Iceberg on Hive 3.1.2 standalone metastore. Iceberg
> issues a lot of lock() calls for commits.
> We hit randomly a strange NPE that fails Iceberg commits.
> {noformat}
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195]
> metastore.RetryingHMSHandler: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy27.lock(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-08-21T11:08:05,665 ERROR [pool-6-thread-195] server.TThreadPoolServer:
> Error occurred during processing of message.
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1903)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1827)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:7217)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at jdk.internal.reflect.GeneratedMethodAccessor52.invoke(Unknown
> Source) ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at com.sun.proxy.$Proxy27.lock(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:18095)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> ~[hadoop-common-3.1.4.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> ~[hive-exec-3.1.2.jar:3.1.2]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-3.1.2.jar:3.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {noformat}
> It seems it's this line, though root cause is not deterined.
> https://github.com/apache/hive/blob/rel/release-3.1.2/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L1903
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
