Yida Wu created IMPALA-11048:
--------------------------------
Summary: load_nested.py fails: File xxx could only be written to 0
of the 1 minReplication nodes
Key: IMPALA-11048
URL: https://issues.apache.org/jira/browse/IMPALA-11048
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.1.0
Reporter: Yida Wu
Saw build failure in impala-cdh-7.1-maint-core:
[https://master-03.jenkins.cloudera.com/job/impala-cdh-7.1-maint-core/74/]
*Error Message*
Error in
/data/jenkins/workspace/impala-cdh-7.1-maint-core/repos/Impala/testdata/bin/create-load-data.sh
at line 48:
*Console Output*
{code:java}
12:10:41 Loading TPC-DS data (logging to
/data/jenkins/workspace/impala-cdh-7.1-maint-core/repos/Impala/logs/data_loading/load-tpcds.log)...
12:10:41 Loading workload 'tpch' using exploration strategy 'core' OK (Took:
6 min 20 sec)
12:17:01 Loading workload 'tpcds' using exploration strategy 'core' OK (Took:
12 min 0 sec)
12:22:41 Loading workload 'functional-query' using exploration strategy
'exhaustive' OK (Took: 22 min 3 sec)
12:32:44 Loading nested parquet data (logging to
/data/jenkins/workspace/impala-cdh-7.1-maint-core/repos/Impala/logs/data_loading/load-nested.log)...
12:32:44 FAILED (Took: 1 min 14 sec)
12:33:58
'/data/jenkins/workspace/impala-cdh-7.1-maint-core/repos/Impala/testdata/bin/load_nested.py
-t tpch_nested_parquet -f parquet/none' failed. Tail of log:
12:33:58 at javax.security.auth.Subject.doAs(Subject.java:422)
12:33:58 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
12:33:58 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2894)
12:33:58
12:33:58 at
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:246)
12:33:58 at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1421)
12:33:58 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:731)
12:33:58 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
12:33:58 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
12:33:58 at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:755)
12:33:58 at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:482)
12:33:58 ... 17 more
12:33:58 Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
File
/test-warehouse/tpch_nested_parquet.db/.hive-staging_hive_2021-12-03_12-33-36_472_3942053844941225298-123/_task_tmp.-ext-10004/_tmp.000000_3
could only be written to 0 of the 1 minReplication nodes. There are 3
datanode(s) running and 3 node(s) are excluded in this operation.
12:33:58 at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2280)
12:33:58 at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
12:33:58 at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2827)
12:33:58 at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:874)
12:33:58 at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:589)
12:33:58 at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
12:33:58 at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
12:33:58 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
12:33:58 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:989)
12:33:58 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:917)
12:33:58 at java.security.AccessController.doPrivileged(Native Method)
12:33:58 at javax.security.auth.Subject.doAs(Subject.java:422)
12:33:58 at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
12:33:58 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2894)
12:33:58
12:33:58 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562)
12:33:58 at org.apache.hadoop.ipc.Client.call(Client.java:1508)
12:33:58 at org.apache.hadoop.ipc.Client.call(Client.java:1405)
12:33:58 at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
12:33:58 at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
12:33:58 at com.sun.proxy.$Proxy13.addBlock(Unknown Source)
12:33:58 at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:523)
12:33:58 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
12:33:58 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
12:33:58 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:33:58 at java.lang.reflect.Method.invoke(Method.java:498)
12:33:58 at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
12:33:58 at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
12:33:58 at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
12:33:58 at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
12:33:58 at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
12:33:58 at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
12:33:58 at
org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:1116)
12:33:58 at
org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1880)
12:33:58 at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1682)
12:33:58 at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:719)
12:33:58 ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1
killedTasks:0, Vertex vertex_1638562107871_0037_1_00 [Map 1] killed/failed due
to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1
killedVertices:0
12:33:58 ERROR in
/data/jenkins/workspace/impala-cdh-7.1-maint-core/repos/Impala/testdata/bin/create-load-data.sh
at line 48:{code}
Saw an error in logs/ee_tests/results/TEST-impala-serial.xml may be related:
{code:java}
ImpalaBeeswaxException:
INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
MESSAGE: AnalysisException: Failed to load metadata for table:
functional.alltypes. Running 'invalidate metadata
functional.alltypes' may resolve this problem.
CAUSED BY: MetaException: Could not connect to meta store using any of the URIs
provided. Most recent failure: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:635)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:422)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient$1.run(RetryingMetaStoreClient.java:188)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:184)
at com.sun.proxy.$Proxy11.getTable(Unknown Source)
at org.apache.impala.catalog.TableLoader.load(TableLoader.java:79)
at
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:245)
at
org.apache.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:242)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 15 more{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
