Esteban,
I totally understand. In this case, I believe it makes my description of the NiFi use case much stronger. As the flow becomes more and more complex, the more and more difficult it becomes to keep an integration test up-to-date (and to build it!) and the easier it becomes to configure via the UI. But if I’m missing something please let me know. Thanks -Mark From: Esteban Aliverti Sent: Tuesday, February 24, 2015 11:15 AM To: [email protected] Thanks for the clarification. I'll try to see what other options do I have. My scenario is not a simple ETL where everything gets resolved using local data. The flow interacts with a bunch of external systems to both, retrieve and send data. I was planning to use mocked services in those cases. Thanks for your time! Regards, XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Esteban Aliverti - Blog @ http://ilesteban.wordpress.com On Tue, Feb 24, 2015 at 4:54 PM, Mark Payne <[email protected]> wrote: > Esteban, > > > > > OK, thanks for clarifying. This notion of how to test your flow has > starting coming up a lot in the last week or so on the dev list. I’ll take > this to be a good thing, meaning that people are really starting to think > about how they can use it! 😊 > > > > > We have actually intentionally avoided creating anything to allow > developers to create unit/integration tests for flows. Creating these sorts > of tests are very common for most ETL types of applications. However, NiFi > was designed specifically so that there was no need to do so. The real-time > command and control that NiFi affords you is very different than that > provided by typical ETL applications. Because of this real-time command and > control, we push operators to build a “test" flow on the production > instance and run the live data through the test flow at the same time that > it is processed by the operational flow. In the test flow, though, the > operator will likely terminate the data at the end, rather than sending to > any other system. This allows the operator to verify that everything is in > good shape with the test flow. Once it has been determined that everything > is in good shape, the operator should then enable sending the data to the > appropriate destinations. This tends to provide much faster time-to-market > of flow changes. > > > Over the past several days there has been another thread discussing the > notion of testing a flow. This thread can be viewed at > http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-td787.html#a811 > > > If you have specific questions/concerns, I’d be happy to discuss in more > detail. > > > Thanks > > -Mark > > > > > > > > > > From: Esteban Aliverti > Sent: Tuesday, February 24, 2015 10:52 AM > To: [email protected] > > > > > > Thanks for the prompt response Mark. > My idea was to have a NiFi workflow defined and to be able to run some > automated tests on it. I thought that maybe there was a way to start an > embedded version of NiFi programatically and to interact with it. > I did see the TestRunner class and I even used it for some of the unit > tests of some custom processors I've implemented. TestRunner seems to be > exactly what I'm looking for, but instead of being focused in a single > processor, I need something that allows me to run multiple processors that > compose a flow. > I do agree that each individual processor must be tested using a > TestRunner, but I was planning on doing something more like an integration > test of an entire flow. > > Regards, > > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > Esteban Aliverti > - Blog @ http://ilesteban.wordpress.com > > On Tue, Feb 24, 2015 at 4:42 PM, Mark Payne <[email protected]> wrote: > > > Esteban, > > > > > > NiFi does not allow you to run it embedded. However, it does provide a > > “Mock Framework” that we use very heavily for doing unit testing of NiFi > > components. > > > > > > Can you explain your use case a bit further? What exactly are you trying > > to unit test? > > > > > > Thanks > > > > -Mark > > > > > > > > > > > > > > From: Esteban Aliverti > > Sent: Tuesday, February 24, 2015 8:34 AM > > To: [email protected] > > > > > > > > > > > > Hello, > > My name is Esteban Aliverti and I'm currently trying to use NiFi in one > of > > my company's projects. The idea is to use NiFi behind a java API. > > What I was wondering is if there's any way to run NiFi embedded so I can > > easily build unit tests without having to start a separate NiFi server > > before I run the tests. Another important thing is that each test should > be > > self-contained and thus I would like them to always run in a "clean" NiFi > > instance. > > What I was thinking was a way to programatically configure and start NiFi > > and maybe import a workflow from a template definition I can create > > beforehand. > > Is this currently possible? If not, any suggestion? > > > > Regards, > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > > > Esteban Aliverti > > - Blog @ http://ilesteban.wordpress.com > > >
