Thanks Joe. I thought NiFi Clusters, or Clustered NiFis are used for HA, Load Balancing, and Scalability, but it seems like it can use for modularization.
On Tue, Feb 16, 2016 at 3:45 PM, Joe Witt <[email protected]> wrote: > Jeff > > Clustered NiFis (multiple nodes) is quite common. You can have > hundreds of processors representing a quite large number of distinct > flows on there. Templates are the mechanism by which flows can be > shared among clusters and between dev/prod environments. We have some > important issues to tackle to make templates as powerful as they > should be such as having environment variable mapping and > consistent/repeatable serialization which lends to better version > control of templates. > > Thanks > Joe > > On Mon, Feb 15, 2016 at 9:39 PM, Jeff - Data Bean Australia > <[email protected]> wrote: > > Thanks Joe for the clarification. And it does make sense to me with > runtime > > reliability and change requirement in enterprise environment. > > > > I know that there are solutions that put multiple NiFi instances working > > together. How common a solution like this in real world? Should we, > > generally speaking, prefer simple flow.xml configuration and combine NiFi > > instances together for more complex scenarios? > > > > On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <[email protected]> wrote: > >> > >> Jeff, > >> > >> Regarding hot deploy of new dataflows: > >> Absolutely. The ability to have interactive command and control is a > >> key feature/concept of NiFi so you can certainly create, alter, remove > >> dataflows while the systems is running by design. This isn't just for > >> a person controlling the flow through UI but also for external > >> systems to have automated interactions with NiFi through its REST API > >> which can do things like change the flow, alter priorities, etc.. > >> > >> Regarding hot deploy of new code/extensions: > >> We've avoided live deploy of new extensions to this point. Largely > >> due to the understanding that while adding new extensions at runtime > >> is pretty doable it is less reliable/clear to change/update. That > >> said, we're trending toward this registry and it would back versions > >> of extensions at which point maybe this becomes more reasonable. > >> > >> Anyway, in the mean time one option may be the fact that the > >> ExecuteScript,InvokeScriptedProcessor processors do support live > >> alteration of the code behind them. While clearly not a complete > >> solution this may help with some of your cases. > >> > >> Thanks > >> Joe > >> > >> > >> > >> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia > >> <[email protected]> wrote: > >> > Here is my use case: I have a quite complex Data Flow system > implemented > >> > by > >> > NiFi and keep streaming data to another system. There is part of the > >> > system > >> > I might want to update on-the-fly without turning down the whole NiFi > >> > platform. And occasionally I might want to add a new processor, or > >> > upgrade > >> > an existing one, then bring up this processor right away. > >> > > >> > Can I do it in NiFi now? > >> > > >> > Thanks, > >> > Jeff > >> > > >> > -- > >> > Data Bean - A Big Data Solution Provider in Australia. > > > > > > > > > > -- > > Data Bean - A Big Data Solution Provider in Australia. > -- Data Bean - A Big Data Solution Provider in Australia.
