Igor,
Your thoughts are correct, and without any additional configuration, the
GetTwitter processor would run on both nodes. The way to avoid this is to
select the "On primary node" scheduling strategy which would only have the
processor run on whichever node is currently primary.
PutHDFS has similar semantics but these would likely be desired. Consider
where data is partitioned across each of the nodes. PutHDFS would then
need to run on each node to ensure the data is delivered to HDFS. The
property you list is that of where the data should land on the configured
HDFS instance. Often times this is done via Expression Language (EL) to
get the familiar time slicing of resources when persisted such as
${now():format('yyyy/MM/dd/HH')}. You could additionally have directory
structure that mirrors the data making use of attributes the files may have
gained as they made their way through your flow or an UpdateAttribute to
set a property, such as "hadoop.dest.dir", that is used by the final
PutHDFS property to give a dynamic location on a per FlowFile basis.
Let us know if you have additional questions or if things are unclear.
--aldrin
On Sun, May 1, 2016 at 1:20 PM, Igor Kravzov <[email protected]> wrote:
> If I understand correctly in cluster mode the same dataflow runs on all
> the notes.
> So let's say I have a simple dataflow with GetTwitter and PutHDFS
> processors. And one NCM + 2 nodes.
> Does it actually that mean the GetTwitter will be called independently and
> potentially simultaneously on each node and there may be duplicate results?
> How about PutHDFS processor? To where "hadoop configuration resources"
> "parent HDFS directory" should point to in each node?
>
