Niliria opened a new issue, #4740: URL: https://github.com/apache/iceberg/issues/4740
So I'm just following the [Getting Started](https://iceberg.apache.org/getting-started/) Apache-Hive-2.1.0 Apache-Hadoop-2.7.3 iceberg-hive-runtime-0.12.1.jar Metastore of Hive is deployed on the local Mysql server When I perform ``` SET iceberg.engine.hive.enabled=true; CREATE DATABASE IF NOT EXISTS iceberg_db; ADD JAR /opt/software/hive/lib/iceberg-hive-runtime-0.12.1.jar; CREATE TABLE iceberg_db.test_3 ( id bigint, name string ) STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' ``` Job exception: ``` 2022-05-10T16:30:54,595 INFO [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Executing command(queryId=root_20220510163054_4db7c534-2f5d-47db-b515-618f64aa0ecc): CREATE TABLE iceberg_db.test_3 ( id bigint, name string ) STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' 2022-05-10T16:30:54,595 INFO [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Starting task [Stage-0:DDL] in serial mode 2022-05-10T16:30:54,595 INFO [HiveServer2-Background-Pool: Thread-15666]: plan.CreateTableDesc (:()) - Use StorageHandler-supplied org.apache.iceberg.mr.hive.HiveIcebergSerDe for table test_3 2022-05-10T16:30:54,596 INFO [HiveServer2-Background-Pool: Thread-15666]: exec.DDLTask (:()) - creating table iceberg_db.test_3 on null 2022-05-10T16:30:54,641 WARN [HiveServer2-Background-Pool: Thread-15666]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.metastore.event.db.notification.api.auth does not exist 2022-05-10T16:30:54,641 WARN [HiveServer2-Background-Pool: Thread-15666]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist 2022-05-10T16:30:54,649 INFO [ForkJoinPool.commonPool-worker-0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 8 2022-05-10T16:30:54,658 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083 2022-05-10T16:30:54,659 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Opened a connection to metastore, current connections: 9 2022-05-10T16:30:54,661 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Connected to metastore. 2022-05-10T16:30:54,666 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"long"},{"id":1,"name":"name","required":false,"type":"string"}]} 2022-05-10T16:30:54,667 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"long"},{"id":1,"name":"name","required":false,"type":"string"}]} 2022-05-10T16:30:54,819 WARN [HiveServer2-Background-Pool: Thread-15666]: metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150) at com.sun.proxy.$Proxy22.lock(Unknown Source) at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76) at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430) at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226) at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165) at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78) at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165) at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-05-10T16:30:55,819 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Closed a connection to metastore, current connections: 8 2022-05-10T16:30:55,819 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083 2022-05-10T16:30:55,819 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Opened a connection to metastore, current connections: 9 2022-05-10T16:30:55,820 INFO [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Connected to metastore. 2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: exec.DDLTask (:()) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:297) at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165) at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78) at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165) at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830) ... 22 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150) at com.sun.proxy.$Proxy22.lock(Unknown Source) at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76) at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430) at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226) ... 40 more 2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 2022-05-10T16:30:55,825 INFO [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Completed executing command(queryId=root_20220510163054_4db7c534-2f5d-47db-b515-618f64aa0ecc); Time taken: 1.23 seconds 2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: operation.Operation (:()) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90) at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235) ... 11 more Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3 at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:297) at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165) at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78) at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165) at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254) at com.sun.proxy.$Proxy22.createTable(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830) ... 22 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150) at com.sun.proxy.$Proxy22.lock(Unknown Source) at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76) at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430) at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226) ... 40 more ``` <img width="1220" alt="image" src="https://user-images.githubusercontent.com/39268464/167586577-e499920a-d3c2-4156-844c-8551563e5098.png"> The metadata folder is created on HDFS; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
