I think one of the things here that may be confusing people is that we are talking about “testing” here in general terms. There are really two types of testing that are relevant here: testing of code (new nars), and testing of dataflow configuration.
To test a new dataflow configuration, I would create the test alongside the production flow on the production instance. When you are happy with what you see, you can switch the production flow to the new one and kill off the old flow. To test new code, as you are discussing here, I would run the new code in a test instance of NiFi. I would then have the production instance push a copy of the production data (possibly a filtered subset of the data or an obfuscated version of the data, as requirements may dictate) to the test instance, and verify that the code works well. This is the approach that I discussed in the blog post. Thanks -Mark From: Mark Payne Sent: Sunday, February 22, 2015 8:00 PM To: [email protected] Bryan, This is largely what I was trying to touch on in my last response to Toivo. It’s also what I was touching on in the last blog that I wrote: https://blogs.apache.org/nifi/entry/say_good_bye_to_canned This is largely how I test all of my software at my “day job.” I spawn a duplicate feed to push to a test instance. That duplicate feed could then be modified/obfuscated as needed for the test cluster. Or you could filter out only part of it to send to the test cluster. Does this make sense? Or am I just confusing people? 😊 From: Bryan Bende Sent: Sunday, February 22, 2015 7:44 PM To: [email protected] What if you are making significant changes to a custom processor? Using Toivo's example, if there is a custom processor for inserting to a relational database, and the graph has one instance inserting to a production database, and another inserting to a test database, is there a good way to test your latest Nar and not affect the production part of the flow? -Bryan On Sun, Feb 22, 2015 at 5:59 PM, Corey Flowers <[email protected]> wrote: > Good afternoon Toivo, > > I work nifi operations/integration daily, on very vital datasets, > and can tell you that we too had to change the views and procedures of our > customers/leadership to accept this type of thinking. NIFI is a step > forward in the evolution of software development not a software to be > placed in previous software development methodologies. Our team has had > this fight numerous times, from canned data, to test environments to > upgrading policies. We even had one customer tell us, we needed a 17 day > notice just to restart NIFI because all previous versions of their software > had to be started in a sequence, previously took 2 hours to start back up > and this was their current operating posture. In our daily activities our > operations personnel work hand in hand with the developers to integrate new > processors into production environments, usually by cloning production data > into grouped development processors on the production graph. This allows us > to expedite the integration process, save money on building test > environments and also allows us to see real load on the production > suite/cluster. There is no doubt, this is not only a change in development > processes but also the mindsets around software development in general. I > can only assure you the fight is well worth it in the end. > > Corey > > On Sun, Feb 22, 2015 at 4:46 PM, Toivo Adams <[email protected]> > wrote: > > > Joe, > > > > Thank you for explanation. > > > > I hope I understand NiFi main idea. > > And it's really well thought and implemented. > > I like it very much. > > > > But. > > “Ultimately the idea of production versus test often means fairly large > > protracted cycles from idea to production outcome.” > > Welcome to my world. > > I work for financial institution. > > > > Maybe I am naive but I do see NiFi as general way how to do software > > development in many business domains. > > I hate to hear NiFi is not usable in financial institution. > > > > I've seen a lot of different business applications. > > One of worst of them are EJB style monolithic web applications. > > I have learned that software should be implemented as side-effect free > > components which will do only one thing but do it very well. Software > > development is very costly and reusing components is the key to > > keep development cost at reasonable level. > > Also debugging and scalability is much simple using such components. > > So NiFi is perfect fit. > > > > IDEAL world > > Development and operations are separated. > > Almost none are allowed to see live data because it contains highly > > sensitive customer data. > > Breaking rules may lead immediate firing. (this is not theory, it has > > happen > > few times) > > So development must use scrambled test data. > > Before even single simple change can be applied to Live system, strict > > change management procedure must be pass through. Yes, this takes usually > > 1-2 days when everything is OK. > > And deploy will be done by operations (not by developers). > > Positive is what all changes to live systems are recorded and anytime > roll > > back can be done quickly (s*it happens). > > When whatever new application or change is in live, it might be running > > many > > years without any efforts from development team. > > > > REAL world > > Sometimes something still goes wrong whatever the reason is. > > Usually monitoring will get alert and will forward problem to > predetermined > > person(persons) > > When operations (administrations) are unable to resolve problem this will > > end up to some developer leader desk. In this case developer is > authorized > > to be use live data to solve problem quickly. > > Now NiFi can again be very, very valuable. > > > > Summary > > I want to use NiFi but at the same time I must follow our strict > test/live > > environment rules. > > Or NiFi would not be accepted at all. > > > > > > Thanks > > toivo > > > > > > > > > > -- > > View this message in context: > > > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-tp787p801.html > > Sent from the Apache NiFi (incubating) Developer List mailing list > archive > > at Nabble.com. > > > > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 > [email protected] > > -- This account not approved for unencrypted proprietary information -- >
