Hello, I have no idea how AWS works, but most likely what is happening is the Hadoop client in NiFi asks the name node to write a file, and the name node then responds with the data nodes to write to, but it is responding with the private IPs/hostnames of the data nodes which you can't reach from your windows machines.
You would probably have the same problem if you tried installing the hadoop client on your windows machine and issue commands like "hadoop fs -ls /". -Bryan On Thu, Apr 19, 2018 at 1:55 AM, Sudheer Nulu <[email protected]> wrote: > Hi Team, > > > > I have recently started exploring Apache nifi to implement in our upcoming > projects. > > > > The issue is: > > > > I have nifi installed on my windows machine to transfer the data from oracle > database to aws instance hdfs layer by using putHDFS. files are transferred > to hdfs with 0 Bytes of memory.(Attached the snapshot for reference) > > > > I was able to place files using putsftp in aws instance without any > issue,but failed to do it in hdfs layer. > > > > > > In logs I could see this: > > > > <logs> > > WARN [Thread-5441] org.apache.hadoop.hdfs.DFSClient DataStreamer Exception > java.nio.channels.UnresolvedAddressException: null > > at sun.nio.ch.Net.checkAddress(Unknown Source) at > sun.nio.ch.SocketChannelImpl.connect(Unknown Source) at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) > > </logs> > > > > Procedure followed: > > I have copied the hdfs-site.xml and core-site.xml from aws instance and > replaced all private hostname with public hostname as I cannot communicate > with private DNS,then I provided those two files as inputs to apache nifi > configuration. > > > > The hdfs layer has full permissions to write to it. > > > > Could someone help on this as I have struck at this point from longtime, > made many changes in hosts file for resolving hostname issue. > > > > Regards, > > Sudheer Nulu > > > > > >
