Thanks Aldrin for the repose. What didn't fully understand from documentation: is automatic fail-over implemented? I would rather configure entire workflow to run "On primary node".
On Sun, May 1, 2016 at 1:31 PM, Aldrin Piri <[email protected]> wrote: > Igor, > > Your thoughts are correct, and without any additional configuration, the > GetTwitter processor would run on both nodes. The way to avoid this is to > select the "On primary node" scheduling strategy which would only have the > processor run on whichever node is currently primary. > > PutHDFS has similar semantics but these would likely be desired. Consider > where data is partitioned across each of the nodes. PutHDFS would then > need to run on each node to ensure the data is delivered to HDFS. The > property you list is that of where the data should land on the configured > HDFS instance. Often times this is done via Expression Language (EL) to > get the familiar time slicing of resources when persisted such as > ${now():format('yyyy/MM/dd/HH')}. You could additionally have directory > structure that mirrors the data making use of attributes the files may have > gained as they made their way through your flow or an UpdateAttribute to > set a property, such as "hadoop.dest.dir", that is used by the final > PutHDFS property to give a dynamic location on a per FlowFile basis. > > Let us know if you have additional questions or if things are unclear. > > --aldrin > > > > On Sun, May 1, 2016 at 1:20 PM, Igor Kravzov <[email protected]> > wrote: > >> If I understand correctly in cluster mode the same dataflow runs on all >> the notes. >> So let's say I have a simple dataflow with GetTwitter and PutHDFS >> processors. And one NCM + 2 nodes. >> Does it actually that mean the GetTwitter will be called independently >> and potentially simultaneously on each node and there may be duplicate >> results? >> How about PutHDFS processor? To where "hadoop configuration resources" >> "parent HDFS directory" should point to in each node? >> > >
