flyrain opened a new issue #2387:
URL: https://github.com/apache/iceberg/issues/2387
I hit this issue, and got it resolved. It should only happen when you
connect a postgres Hive metastore DB. Open this issue in case somebody else hit
it. It is similar to issue #798, but the root case should be a bit different.
Here is the client side stack.
```
scala> spark.sql("CREATE TABLE testdb.student (id INT, name STRING, age INT)
USING ADT PARTITIONED by(age) LOCATION 'file:/tmp/testdb/student'").show
java.lang.RuntimeException: Metastore operation failed for testdb.student
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:204)
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:107)
...
Caused by: org.apache.thrift.TApplicationException: Internal error
processing lock
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3910)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3897)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1955)
at
org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$8(HiveTableOperations.java:277)
at org.apache.iceberg.hive.ClientPool.run(ClientPool.java:54)
at
org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:277)
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:145)
... 72 more
```
Server side stack
```
2021-03-26 18:47:03,240 ERROR thrift.ProcessFunction
(ProcessFunction.java:process(41)) - Internal error processing lock
MetaException(message:Unable to update transaction database
org.postgresql.util.PSQLException: ERROR: index "hl_txnid_index" contains
unexpected zero page at block 0
Hint: Please REINDEX it.
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2455)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2155)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
at
org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303)
at
org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289)
at
org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
at
org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:246)
at
com.jolbox.bonecp.StatementHandle.executeUpdate(StatementHandle.java:497)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:1428)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:422)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5527)
```
To fix it, we need to reindex hl_txnid_index. Here is an example command.
```
psql -U hive -d metastore --no-password -h localhost --command "REINDEX
INDEX hl_txnid_index"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
