Hello, RPG can be used to distribute workload among NiFi nodes. You can use RPG between ListSFTP and FetchSFTP (I assume it's not GetSFTP). But RPG does not help for a flow to wait certain set of FlowFiles to be processed. As for waiting all FlowFiles to go thru PutHDFS processor, Wait/Notify processor might be useful. Though, it may be challenging since ListSFTP does not write an attribute to tell the number of files it listed in a cycle. If you can share directory and file structure you are watching with ListSFTP, more detailed answer may be provided. As Pierre replied on the blog post, the need for waiting is not clear.
Thanks, Koji On Thu, Dec 21, 2017 at 3:49 PM, satish <[email protected]> wrote: > Hi I have followed pierrevillard > <https://pierrevillard.com/2017/02/23/listfetch-pattern-and-remote-process-group-in-apache-nifi/comment-page-1/#comment-1188> > blog to move files from sftp to hdfs. > > I have facing problem in my current workflow which is running on clustered > NiFi. My workflow is LISTSFTP(on > PrimaryNode)->GETSFTP->PUTHDFS->CREATE_HIVE_EXTERNAL_TABLE(on PrimaryNode)-> > INSERT_INTO_MANAGED_TABLE(on PrimaryNode) . Everything is fine until > PUTHDFS. But I have facing problem in creating the external table in hive, > as I need to wait all PUTHDFS processes ( which are spanned across cluster) > have completed before triggering external table processor. How can I achieve > this? > > Whether wait and notify processes can resolve this issue or else I need to > implement using RPG? > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
