simhadri-g commented on PR #5573:
URL: https://github.com/apache/hive/pull/5573#issuecomment-2541977029
This seems to be a genuine Test failure .
I was trying to start Metastore with MYSQL using the latest master branch.
But the metastore is failing to start due to deadlock . (PS: I don't see
this issue in hive 4.0.1. ) .
Please find the stack trace below.
```
2024-12-13T14:05:31,949 INFO [Metastore Scheduled Worker 6]
metastore.HMSHandler: Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@5bd5b219
2024-12-13T14:05:32,935 INFO [Metastore Scheduled Worker 8]
service.AcidOpenTxnsCounterService: Open txn counter ran for 0 seconds.
isAliveCounter: 61
2024-12-13T14:05:41,934 INFO [Metastore Scheduled Worker 8]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:05:51,933 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:01,932 INFO [Metastore Scheduled Worker 6]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:11,930 INFO [Metastore Scheduled Worker 12]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:21,933 INFO [Metastore Scheduled Worker 11]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:31,931 INFO [Metastore Scheduled Worker 2]
service.AcidHouseKeeperService: Starting to run AcidHouseKeeperService
2024-12-13T14:06:31,934 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:31,938 INFO [Metastore Scheduled Worker 2]
functions.CleanTxnToWriteIdTableFunction: Removed 0 rows from TXN_TO_WRITE_ID
with Txn Low-Water-Mark: 1
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.HMSHandler: Opening raw store with implementation class:
org.apache.hadoop.hive.metastore.ObjectStore
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.ObjectStore: RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e, with PersistenceManager:
null will be shutdown
2024-12-13T14:06:31,947 INFO [Metastore Scheduled Worker 12]
metastore.ObjectStore: RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e, with PersistenceManager:
org.datanucleus.api.jdo.JDOPersistenceManager@684f4e2 created in the thread
with id: 51
2024-12-13T14:06:31,949 INFO [Metastore Scheduled Worker 12]
metastore.HMSHandler: Created RawStore:
org.apache.hadoop.hive.metastore.ObjectStore@565c37e
2024-12-13T14:06:33,931 INFO [Metastore Scheduled Worker 3]
service.AcidOpenTxnsCounterService: Open txn counter ran for 0 seconds.
isAliveCounter: 122
2024-12-13T14:06:41,933 INFO [Metastore Scheduled Worker 3]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:06:51,933 INFO [Metastore Scheduled Worker 1]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:07:01,926 INFO [Metastore Scheduled Worker 9]
txn.CompactionTxnHandler: Start to clean empty aborted or committed TXNS
2024-12-13T14:07:02,069 ERROR [Metastore Scheduled Worker 2]
metastore.MaterializationsRebuildLockCleanerTask: Unexpected error in thread:
Metastore Scheduled Worker 2, message: Unable to lock MaterializationRebuild
due to: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1",
"MT_KEY2") VALUES(?, 0)]; Deadlock found when trying to get lock; try
restarting transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
java.lang.RuntimeException: Unable to lock MaterializationRebuild due to:
PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2")
VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting
transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:85)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.MaterializationsRebuildLockCleanerTask.run(MaterializationsRebuildLockCleanerTask.java:65)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_362]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
~[?:1.8.0_362]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
~[?:1.8.0_362]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
~[?:1.8.0_362]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_362]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.springframework.dao.DeadlockLoserDataAccessException:
PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2")
VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting
transaction; nested exception is
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
at
org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:268)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1575)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
... 8 more
Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException:
Deadlock found when trying to get lock; try restarting transaction
at
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1098)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1046)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1371)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1031)
~[mysql-connector-java-8.0.28.jar:8.0.28]
at
org.apache.hive.com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331)
~[spring-jdbc-5.3.39.jar:5.3.39]
at
org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
... 8 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
