Hi, Does anyone know why a datanode might make a request to it's own dfs.datanode.address port (50010 in my case) every 60 seconds?
Specifically, I'm seeing an error like this every minute: > tail -f /var/log/hadoop/hdfs/hadoop-hdfs-datanode-myhost.log 2015-05-20 14:07:41,030 ERROR datanode.DataNode (DataXceiver.java:run(253)) - myhost.mycompany.com:50010:DataXceiver error processing unknown operation src: /127.0.0.1:39352 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:357) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:177) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:109) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:183) at java.lang.Thread.run(Thread.java:745) 2015-05-20 14:07:41,144 WARN mortbay.log (Slf4jLog.java:warn(89)) - EXCEPTION javax.net.ssl.SSLHandshakeException: null cert chain at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266) at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1631) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:176) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) at sun.security.ssl.Handshaker.process_record(Handshaker.java:804) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:723) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) The error itself is likely related to the fact that we have mutual ssl authentication enabled but for the life of me, I don't know where to look in the Hadoop code to see what the *client *is using for its ssl connection configuration. I suspect that the client keystore either isn't being set or that we have to do something special with our ssl certificates to allow mutual ssl connection to AND from 127.0.0.1 (I already tried setting hadoop.ssl.hostname.verifier to DEFAULT_AND_LOCALHOST to no avail). Either way, if somebody knows what DataNode function might involve calling itself, that would be a huge help! Thanks. Hadoop Version = 2.6.0 Java Version = Oracle 1.7.0_67
