Michael Smith created IMPALA-13633:
--------------------------------------
Summary: Unable to use manipulate table if current snapshot is
missing
Key: IMPALA-13633
URL: https://issues.apache.org/jira/browse/IMPALA-13633
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 4.4.1
Reporter: Michael Smith
If the current snapshot file for an Iceberg table is accidentally deleted
(perhaps due to a cleanup script gone haywire), the table becomes unusable.
Operations that seem like they should still work:
* DROP TABLE <table> - appears to remove the table from HMS then sometimes
errors later
* DESCRIBE HISTORY <table> - metadata file tracks snapshot relationships
* ALTER TABLE <table> EXECUTE ROLLBACK(<snapshot id>)
* SELECT <expr> FROM <table> FOR SYSTEM_VERSION AS OF (<snapshot id>)
If the {{current-snapshot-id}} value is missing, loading the table results in
an error and none of the above are possible (need to {{invalidate metadata}}
after the file is deleted)
{code}
NotFoundException: Failed to open input stream for file:
hdfs://localhost:20500/test-warehouse/foo/metadata/snap-9009021153525485665-1-7035498a-5a3b-4e2d-8a21-21054efb6d1e.avro
CAUSED BY: FileNotFoundException: File does not exist:
/test-warehouse/foo/metadata/snap-9009021153525485665-1-7035498a-5a3b-4e2d-8a21-21054efb6d1e.avro
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:738)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:995)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:923)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2901)
CAUSED BY: RemoteException: File does not exist:
/test-warehouse/foo/metadata/snap-9009021153525485665-1-7035498a-5a3b-4e2d-8a21-21054efb6d1e.avro
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:87)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:77)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:159)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:738)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:454)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:995)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:923)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1910)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2901)
{code}
Spark has some ability to continue reading other snapshots. A workaround is
also to directly modify {{current-snapshot-id}} in the latest metadata file,
but that may have unintended consequences.
A similar issue occurs if a file mentioned by the current snapshot is missing
{code}
AnalysisException: org.apache.impala.catalog.TableLoadingException:
IcebergTableLoadingException: Error loading metadata for Iceberg table
hdfs://localhost:20500/test-warehouse/foo
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths
for table default.foo: failed to load 1 paths. Check the catalog server log for
more details.
CAUSED BY: TableLoadingException: IcebergTableLoadingException: Error loading
metadata for Iceberg table hdfs://localhost:20500/test-warehouse/foo
CAUSED BY: TableLoadingException: Loading file and block metadata for 1 paths
for table default.foo: failed to load 1 paths. Check the catalog server log for
more details.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
