Sorry for the late replying. And thanks for the insights which are helpful!
The second answer leads me to another question. I need to automate the process (auto deploying Nifi to remote production server). Searching result comes with using HDP, which looks like Hortonwork specific. https://community.hortonworks.com/articles/58330/automation-to-deploy-hdp-25nifi-10-clusters-runnin.html Is this the only (recommended) way to deploy Nifi? I am looking for a solution e.g. ansible for auto deploying Nifi, and my requirements are basically 1. installing and configuring Nifi, 2. creating flow graph, 3. starting the flow. So generally there won't have manual configuration (open browser, create flow in UI, etc.). How can I achieve this? Thanks > On 03 April 2017 at 20:10 Andy LoPresto <[email protected]> wrote: > > Hi Richard, > > 1. NiFi does not have a defined maximum cluster size. For the best > performance, we usually recommend < 10 nodes per cluster, but no more. If you > have high performance needs, we have generally seen the best results with > multiple smaller clusters than one large one. In this way, you can have > hundreds of nodes processing the data in parallel, but the cluster > administration overhead does not tax a single cluster coordinator to death. > 2. While it is technically possible to create and define a flow.xml.gz > file by hand, this would be incredibly frustrating, as the components and > connections need a high number of defined values and must be validated in > many unique ways. The UI and API allow this to happen in a convenient manner. > If you genuinely wish to define the flow without a UI, take a look at > existing flow.xml.gz files to get an understanding of the flow definition > format. > 3. NiFi can run on small hardware, such as a Raspberry Pi. You may also > be interested in MiNiFi [1], a sub-project of NiFi. MiNiFi is a “headless > agent” tool which is designed to run on lightweight or shared systems and > extend the reach and capabilities of NiFi to the “edge” of data collection. > MiNiFi offers two versions — a Java version [2] which has a high degree of > compatibility with NiFi (many of the native processors are available), and a > C++ version [3] which is extremely compact but has limited processor > definition at this time. MiNiFi may also be a better fit for your “non-UI > workflow”, as the flow can be defined using the GUI of NiFi and then exported > as YAML to the MiNiFi agent, or written directly as YAML if desired. > > [1] https://nifi.apache.org/minifi/index.html > [2] https://github.com/apache/nifi-minifi > [3] https://github.com/apache/nifi-minifi-cpp > > Andy LoPresto > [email protected] mailto:[email protected] > [email protected] mailto:[email protected] > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > On Apr 2, 2017, at 3:45 AM, Richard Hanson > <[email protected] mailto:[email protected] > wrote: > > > > > > I am new to Nifi, and am evaluating it after playing some basic > > functions. Now I have a few questions: > > > > - What is the max cluster size nifi can achieve? > > > > - Is it possible to create workflow without GUI? A bit like travis > > ci .yaml (User creates related workflow file and let nifi execute it (via > > submit or programmatically) > > > > - Is it possible to run embedded nifi? Checking > > http://apache-nifi-developer-list.39713.n7.nabble.com/Possibility-of-running-NiFi-embedded-in-a-test-td820.html > > showing nifi can not run as embedded, but I want to double check for sure > > (not for unit testing). > > > > Thanks > > > > > > > > > > > > > > > > > > > > > >
