Toivo: Congrats and thanks for kicking off a really important thread. This is going to be an important thing for us to understand and communicate effectively. I think several of us have been doing it differently now for a while and so we think and talk about it differently as a result.
I totally accept that the scenario and needs you describe are real. Just want to really understand the details and see if there is a more 'with the grain' approach. I definitely hope NiFi works out to be a really useful tool for your case and if we can find a clean and consistent way to support your scenario great. We're trying to be awesome at the sensor driven use cases. There are lots of cases to suggest that we have strength as a general dataflow solution that but we'll see. So as long as you'll keep working with us to understand we'll explore it and figure it out. We need to be good as a community at both accepting new ideas and also accepting that some things just don't fit well. Keep in mind that our base design being oriented on Flow Based Programming encourages highly cohesive and loosely coupled processes. This focus on cohesion tends to dramatically ease the testing challenges that exist. True scale tests are almost impossible to emulate until you're in the full heat of the production fire. But in any event the FBP model does a great job of helping you knock down the vast majority of the issues through truly effective unit testing. These cohesive elements are also great for reuse as you mentioned previously. For now is the only thing you're requesting that you'd like to see variables that can be substituted? We've talked about doing something like this before not necessarily for replacement purposes but because we wanted people to be able to set a value once and reuse it often. But i could see that supporting this too. I want to really understand your case better before we head down that route. Thanks Joe On Sun, Feb 22, 2015 at 7:57 PM, Mark Payne <[email protected]> wrote: > Bryan, > > > This is largely what I was trying to touch on in my last response to Toivo. > It’s also what I was touching on in the last blog that I wrote: > https://blogs.apache.org/nifi/entry/say_good_bye_to_canned > > > This is largely how I test all of my software at my “day job.” I spawn a > duplicate feed to push to a test instance. That duplicate feed could then be > modified/obfuscated as needed for the test cluster. Or you could filter out > only part of it to send to the test cluster. > > > Does this make sense? Or am I just confusing people? 😊 > > > > > > > > > From: Bryan Bende > Sent: Sunday, February 22, 2015 7:44 PM > To: [email protected] > > > > > > What if you are making significant changes to a custom processor? > > Using Toivo's example, if there is a custom processor for inserting to a > relational database, and the graph has one instance inserting to a > production database, and another inserting to a test database, is there a > good way to test your latest Nar and not affect the production part of the > flow? > > -Bryan > > > On Sun, Feb 22, 2015 at 5:59 PM, Corey Flowers <[email protected]> > wrote: > >> Good afternoon Toivo, >> >> I work nifi operations/integration daily, on very vital datasets, >> and can tell you that we too had to change the views and procedures of our >> customers/leadership to accept this type of thinking. NIFI is a step >> forward in the evolution of software development not a software to be >> placed in previous software development methodologies. Our team has had >> this fight numerous times, from canned data, to test environments to >> upgrading policies. We even had one customer tell us, we needed a 17 day >> notice just to restart NIFI because all previous versions of their software >> had to be started in a sequence, previously took 2 hours to start back up >> and this was their current operating posture. In our daily activities our >> operations personnel work hand in hand with the developers to integrate new >> processors into production environments, usually by cloning production data >> into grouped development processors on the production graph. This allows us >> to expedite the integration process, save money on building test >> environments and also allows us to see real load on the production >> suite/cluster. There is no doubt, this is not only a change in development >> processes but also the mindsets around software development in general. I >> can only assure you the fight is well worth it in the end. >> >> Corey >> >> On Sun, Feb 22, 2015 at 4:46 PM, Toivo Adams <[email protected]> >> wrote: >> >> > Joe, >> > >> > Thank you for explanation. >> > >> > I hope I understand NiFi main idea. >> > And it's really well thought and implemented. >> > I like it very much. >> > >> > But. >> > “Ultimately the idea of production versus test often means fairly large >> > protracted cycles from idea to production outcome.” >> > Welcome to my world. >> > I work for financial institution. >> > >> > Maybe I am naive but I do see NiFi as general way how to do software >> > development in many business domains. >> > I hate to hear NiFi is not usable in financial institution. >> > >> > I've seen a lot of different business applications. >> > One of worst of them are EJB style monolithic web applications. >> > I have learned that software should be implemented as side-effect free >> > components which will do only one thing but do it very well. Software >> > development is very costly and reusing components is the key to >> > keep development cost at reasonable level. >> > Also debugging and scalability is much simple using such components. >> > So NiFi is perfect fit. >> > >> > IDEAL world >> > Development and operations are separated. >> > Almost none are allowed to see live data because it contains highly >> > sensitive customer data. >> > Breaking rules may lead immediate firing. (this is not theory, it has >> > happen >> > few times) >> > So development must use scrambled test data. >> > Before even single simple change can be applied to Live system, strict >> > change management procedure must be pass through. Yes, this takes usually >> > 1-2 days when everything is OK. >> > And deploy will be done by operations (not by developers). >> > Positive is what all changes to live systems are recorded and anytime >> roll >> > back can be done quickly (s*it happens). >> > When whatever new application or change is in live, it might be running >> > many >> > years without any efforts from development team. >> > >> > REAL world >> > Sometimes something still goes wrong whatever the reason is. >> > Usually monitoring will get alert and will forward problem to >> predetermined >> > person(persons) >> > When operations (administrations) are unable to resolve problem this will >> > end up to some developer leader desk. In this case developer is >> authorized >> > to be use live data to solve problem quickly. >> > Now NiFi can again be very, very valuable. >> > >> > Summary >> > I want to use NiFi but at the same time I must follow our strict >> test/live >> > environment rules. >> > Or NiFi would not be accepted at all. >> > >> > >> > Thanks >> > toivo >> > >> > >> > >> > >> > -- >> > View this message in context: >> > >> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-tp787p801.html >> > Sent from the Apache NiFi (incubating) Developer List mailing list >> archive >> > at Nabble.com. >> > >> >> >> >> -- >> Corey Flowers >> Vice President, Onyx Point, Inc >> (410) 541-6699 >> [email protected] >> >> -- This account not approved for unencrypted proprietary information -- >>
