If that's the case, you may be able to fix that by adding the following property to your hdfs-site.xml:
dfs.client.use.datanode.hostname = true It should cause the hostnames of the data nodes to be returned to the client, rather than private/inaccessible IPs. Regards, Matt On Thu, Apr 19, 2018 at 8:22 AM, Bryan Bende <[email protected]> wrote: > Hello, > > I have no idea how AWS works, but most likely what is happening is the > Hadoop client in NiFi asks the name node to write a file, and the name > node then responds with the data nodes to write to, but it is > responding with the private IPs/hostnames of the data nodes which you > can't reach from your windows machines. > > You would probably have the same problem if you tried installing the > hadoop client on your windows machine and issue commands like "hadoop > fs -ls /". > > -Bryan > > > On Thu, Apr 19, 2018 at 1:55 AM, Sudheer Nulu <[email protected]> wrote: >> Hi Team, >> >> >> >> I have recently started exploring Apache nifi to implement in our upcoming >> projects. >> >> >> >> The issue is: >> >> >> >> I have nifi installed on my windows machine to transfer the data from oracle >> database to aws instance hdfs layer by using putHDFS. files are transferred >> to hdfs with 0 Bytes of memory.(Attached the snapshot for reference) >> >> >> >> I was able to place files using putsftp in aws instance without any >> issue,but failed to do it in hdfs layer. >> >> >> >> >> >> In logs I could see this: >> >> >> >> <logs> >> >> WARN [Thread-5441] org.apache.hadoop.hdfs.DFSClient DataStreamer Exception >> java.nio.channels.UnresolvedAddressException: null >> >> at sun.nio.ch.Net.checkAddress(Unknown Source) at >> sun.nio.ch.SocketChannelImpl.connect(Unknown Source) at >> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) >> >> </logs> >> >> >> >> Procedure followed: >> >> I have copied the hdfs-site.xml and core-site.xml from aws instance and >> replaced all private hostname with public hostname as I cannot communicate >> with private DNS,then I provided those two files as inputs to apache nifi >> configuration. >> >> >> >> The hdfs layer has full permissions to write to it. >> >> >> >> Could someone help on this as I have struck at this point from longtime, >> made many changes in hosts file for resolving hostname issue. >> >> >> >> Regards, >> >> Sudheer Nulu >> >> >> >> >> >>
