Jeff Clustered NiFis (multiple nodes) is quite common. You can have hundreds of processors representing a quite large number of distinct flows on there. Templates are the mechanism by which flows can be shared among clusters and between dev/prod environments. We have some important issues to tackle to make templates as powerful as they should be such as having environment variable mapping and consistent/repeatable serialization which lends to better version control of templates.
Thanks Joe On Mon, Feb 15, 2016 at 9:39 PM, Jeff - Data Bean Australia <[email protected]> wrote: > Thanks Joe for the clarification. And it does make sense to me with runtime > reliability and change requirement in enterprise environment. > > I know that there are solutions that put multiple NiFi instances working > together. How common a solution like this in real world? Should we, > generally speaking, prefer simple flow.xml configuration and combine NiFi > instances together for more complex scenarios? > > On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <[email protected]> wrote: >> >> Jeff, >> >> Regarding hot deploy of new dataflows: >> Absolutely. The ability to have interactive command and control is a >> key feature/concept of NiFi so you can certainly create, alter, remove >> dataflows while the systems is running by design. This isn't just for >> a person controlling the flow through UI but also for external >> systems to have automated interactions with NiFi through its REST API >> which can do things like change the flow, alter priorities, etc.. >> >> Regarding hot deploy of new code/extensions: >> We've avoided live deploy of new extensions to this point. Largely >> due to the understanding that while adding new extensions at runtime >> is pretty doable it is less reliable/clear to change/update. That >> said, we're trending toward this registry and it would back versions >> of extensions at which point maybe this becomes more reasonable. >> >> Anyway, in the mean time one option may be the fact that the >> ExecuteScript,InvokeScriptedProcessor processors do support live >> alteration of the code behind them. While clearly not a complete >> solution this may help with some of your cases. >> >> Thanks >> Joe >> >> >> >> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia >> <[email protected]> wrote: >> > Here is my use case: I have a quite complex Data Flow system implemented >> > by >> > NiFi and keep streaming data to another system. There is part of the >> > system >> > I might want to update on-the-fly without turning down the whole NiFi >> > platform. And occasionally I might want to add a new processor, or >> > upgrade >> > an existing one, then bring up this processor right away. >> > >> > Can I do it in NiFi now? >> > >> > Thanks, >> > Jeff >> > >> > -- >> > Data Bean - A Big Data Solution Provider in Australia. > > > > > -- > Data Bean - A Big Data Solution Provider in Australia.
