fhj123456789 opened a new issue, #23945: URL: https://github.com/apache/doris/issues/23945
### Discussed in https://github.com/apache/doris/discussions/23522 <div type='discussions-op-text'> <sup>Originally posted by **fhj123456789** August 26, 2023</sup> 我的Doris 和 hadoop 集群是分开安装的，现在出现了一个问题，当我使用SparkLoad的时候，Doris会向HDFS写一些文件，现在的情况是HDFS上确实已经写入的了些文件，但是Spark任务无法调用成功，报错信息为HDFS connection refuesd，我查看日志发现，doris与hdfs 建立的pipieline 第一个是本机，但是我的本机并不是datanode节点，我该怎么解决呢</div> pipeline = [DatanodeInfoWithStorage[doris_extends:50010,DS-e2bda4c9-d6c7-4ec2-ac51-801141db9128,DISK], DatanodeInfoWithStorage[data_node1:50010,DS-5a8d2f9b-c63f-404b-bda4-6582242c41ba,DISK], DatanodeInfoWithStorage[data_node2:50010,DS-44319f5b-1ad8-41f6-82de-a5945f6ac3dc,DISK]] [DEBUG] 2023-08-26 13:38:06,907 method:org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:255) Connecting to datanode doris_extends:50010 [INFO ] 2023-08-26 13:38:06,908 method:org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1763) Exception in createBlockOutputStream java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532) at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:259) at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1699) at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1655) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:710) [WARN ] 2023-08-26 13:38:06,908 method:org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1658) 这是我的报错信息，IP信息我以集群节点名代替。doris节点与hdfs节点是分开部署的，但是每次获取pipiline的时候，都是把doris节点作为首个datanode节点导致创建Stream错误，我也尝试过把hdfs-site.xml 放到extends和be以及fe的目录下，但是都没有解决这个问题。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
