Hi, We are creating processgroups to transfer data from File system to HDFS/HBASE (separate process groups for each destination).
Simple Example: FileSystem to HDFS process group: We have ListFile, FetchFile and PutHDFS process to transfer data from File system to HDFS. When “ListFile” processor’s execution mode is configured with “All Nodes”, there is an error saying lease expired due to race condition. What I understood from exploration is, all nodes are trying to run “listFile” on target directory and all nodes in the cluster are trying to transfer same files, when processing is complete by one of the node, other node is throwing this exception message. In resolving that, we set “ListFIle” execution mode to run in “PrimaryNode”, racing issue is resolved, but now all the files in target directory are processed by only one node in the cluster(no distribution). Data transfer takes longer duration for large files. Further, we added “RPG” processor between “ListFile” and “FetchFile”, so that ListFile is executed by only one node and output of listfile is provided to RPG, further “FetchFile” and “PutHDFS” works in real cluster mode by processing each file by separate nodes in the cluster. This approach is applied for all type of file repositories (SFTP, NFS and HDFS). It works well so far, but I would like to get experts opinion here to make sure we are not missing any. Kind of surprised why NiFi is not working as expected with “ListFile” running in cluster mode, as a user I expect NiFi intelligently decide a node to get list of files and FetchFile to work in cluster mode to distribute the data transfer. Thanks, Ravi Papisetti
