massdosage commented on pull request #1407:
URL: https://github.com/apache/iceberg/pull/1407#issuecomment-694186791
Hey @pvary, apologies for delays in looking at this. Today I tried this out
"for real" on a Hive cluster we have running on EMR where I've been doing my
InputFormat tests. I built a hive-runtime jar from this PR and added to the
classpath of the Hive shell. I then ran:
`CREATE EXTERNAL TABLE bdp.iceberg_write_table STORED BY
'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION
'hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/table_a';`
However this fails with the following exception that I see in the Hive logs:
```
2020-09-17T11:59:02,006 INFO [main([])]: conf.HiveConf
(HiveConf.java:getLogIdVar(3957)) - Using the default value passed in for log
id: 0e8d5f3d-e100-4e3b-bae1-f272b504b04a
2020-09-17T11:59:02,007 INFO [main([])]: session.SessionState
(SessionState.java:updateThreadName(421)) - Updating thread name to
0e8d5f3d-e100-4e3b-bae1-f272b504b04a main
2020-09-17T11:59:02,007 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:compile(429)) - Compiling
command(queryId=hadoop_20200917115902_126001a6-6193-4e7e-8def-345036e43056):
CREATE TABLE bdp.iceberg_write_table STORED BY
'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION
'hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table'
TBLPROPERTIES ('iceberg.mr.write.file.format'='parquet')
2020-09-17T11:59:02,010 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: parse.CalcitePlanner (SemanticAnalyzer.java:analyzeInternal(11150))
- Starting Semantic Analysis
2020-09-17T11:59:02,011 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: parse.CalcitePlanner
(SemanticAnalyzer.java:analyzeCreateTable(11896)) - Creating table
bdp.iceberg_write_table position=13
2020-09-17T11:59:02,043 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:compile(518)) - Semantic Analysis Completed
2020-09-17T11:59:02,044 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:getSchema(290)) - Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
2020-09-17T11:59:02,044 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:compile(642)) - Completed compiling
command(queryId=hadoop_20200917115902_126001a6-6193-4e7e-8def-345036e43056);
Time taken: 0.037 seconds
2020-09-17T11:59:02,044 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:checkConcurrency(210)) - Concurrency mode is
disabled, not creating a lock manager
2020-09-17T11:59:02,044 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:execute(1735)) - Executing
command(queryId=hadoop_20200917115902_126001a6-6193-4e7e-8def-345036e43056):
CREATE TABLE bdp.iceberg_write_table STORED BY
'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler' LOCATION
'hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table'
TBLPROPERTIES ('iceberg.mr.write.file.format'='parquet')
2020-09-17T11:59:02,091 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:launchTask(2181)) - Starting task
[Stage-0:DDL] in serial mode
2020-09-17T11:59:02,091 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: plan.CreateTableDesc (CreateTableDesc.java:toTable(717)) - Use
StorageHandler-supplied org.apache.iceberg.mr.hive.HiveIcebergSerDe for table
iceberg_write_table
2020-09-17T11:59:02,097 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: exec.DDLTask (DDLTask.java:createTable(4324)) - creating table
bdp.iceberg_write_table on
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table
2020-09-17T11:59:02,116 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: mr.Catalogs (Catalogs.java:loadCatalog(114)) - Catalog is not
configured
2020-09-17T11:59:02,124 WARN [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: hadoop.HadoopTableOperations
(HadoopTableOperations.java:readVersionHint(292)) - Error reading version hint
file
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table/metadata/version-hint.text
java.io.FileNotFoundException: File does not exist:
/hiveberg/iceberg_write_table/metadata/version-hint.text
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:692)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_212]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_212]
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:849)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:836)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:825)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:286)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1064)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:337)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:790)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.readVersionHint(HadoopTableOperations.java:287)
~[?:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.refresh(HadoopTableOperations.java:94)
~[?:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.current(HadoopTableOperations.java:75)
~[?:?]
at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:79)
~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:77) ~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:65) ~[?:?]
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:66)
~[?:?]
at
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:449)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:624)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:836)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:872)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4356)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
~[hadoop-common-2.8.5-amzn-4.jar:?]
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/hiveberg/iceberg_write_table/metadata/version-hint.text
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:692)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at com.sun.proxy.$Proxy29.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:259)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at com.sun.proxy.$Proxy30.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:847)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
... 49 more
2020-09-17T11:59:02,144 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: conf.HiveConf (HiveConf.java:getLogIdVar(3957)) - Using the default
value passed in for log id: 0e8d5f3d-e100-4e3b-bae1-f272b504b04a
2020-09-17T11:59:02,144 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: session.SessionState (SessionState.java:resetThreadName(432)) -
Resetting thread name to main
2020-09-17T11:59:02,128 ERROR [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: metadata.Table (Table.java:getColsInternal(644)) - Unable to get
field from serde: org.apache.iceberg.mr.hive.HiveIcebergSerDe
org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at
location:
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table
at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:82)
~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:77) ~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:65) ~[?:?]
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:66)
~[?:?]
at
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:449)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:641)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:624)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:836)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:872)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4356)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
~[hadoop-common-2.8.5-amzn-4.jar:?]
2020-09-17T11:59:02,128 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: mr.Catalogs (Catalogs.java:loadCatalog(114)) - Catalog is not
configured
2020-09-17T11:59:02,129 WARN [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: hadoop.HadoopTableOperations
(HadoopTableOperations.java:readVersionHint(292)) - Error reading version hint
file
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table/metadata/version-hint.text
java.io.FileNotFoundException: File does not exist:
/hiveberg/iceberg_write_table/metadata/version-hint.text
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:692)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_212]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_212]
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:849)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:836)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:825)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:325)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:286)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:270)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1064)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:337)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:790)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.readVersionHint(HadoopTableOperations.java:287)
~[?:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.refresh(HadoopTableOperations.java:94)
~[?:?]
at
org.apache.iceberg.hadoop.HadoopTableOperations.current(HadoopTableOperations.java:75)
~[?:?]
at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:79)
~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:77) ~[?:?]
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:65) ~[?:?]
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:66)
~[?:?]
at
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:449)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:838)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:872)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4356)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
~[hive-exec-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
~[hive-cli-2.3.5-amzn-0.jar:2.3.5-amzn-0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
~[hadoop-common-2.8.5-amzn-4.jar:?]
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist:
/hiveberg/iceberg_write_table/metadata/version-hint.text
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:72)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:62)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:152)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1827)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:692)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:381)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at com.sun.proxy.$Proxy29.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:259)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_212]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_212]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
~[hadoop-common-2.8.5-amzn-4.jar:?]
at com.sun.proxy.$Proxy30.getBlockLocations(Unknown Source) ~[?:?]
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:847)
~[hadoop-hdfs-client-2.8.5-amzn-4.jar:?]
... 47 more
2020-09-17T11:59:02,143 ERROR [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: exec.DDLTask (DDLTask.java:failed(639)) -
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at
location:
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:867)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:872)
at
org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4356)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:354)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: org.apache.iceberg.exceptions.NoSuchTableException: Table does
not exist at location:
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table
at org.apache.iceberg.hadoop.HadoopTables.load(HadoopTables.java:82)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:77)
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:65)
at
org.apache.iceberg.mr.hive.HiveIcebergSerDe.initialize(HiveIcebergSerDe.java:66)
at
org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:449)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:838)
... 22 more
2020-09-17T11:59:02,143 ERROR [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (SessionState.java:printError(1126)) - FAILED: Execution
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist at
location:
hdfs://ip-redacted.us-west-2.compute.internal:8020/hiveberg/iceberg_write_table
2020-09-17T11:59:02,143 INFO [0e8d5f3d-e100-4e3b-bae1-f272b504b04a
main([])]: ql.Driver (Driver.java:execute(2050)) - Completed executing
command(queryId=hadoop_20200917115902_126001a6-6193-4e7e-8def-345036e43056);
Time taken: 0.099 seconds
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
