Toivo,
Please consider the following as an alternative to using substitution
of variables for things like PROD db and TEST db:
Part of the power of NiFi is the real-time command and control
enabling immediate fine-grained modification but also clear and
concise visualization of what is happening in the flow. There are
often little variations between a flow in production and a flow in
test environments because it is indeed so difficult to emulate it all
perfectly. So, please consider setting up your flow such that you use
a RouteOnAttribute processor to send data to your 'production'
database or your 'test' database based on an environment variable.
So your flow could be something like
<do stuff> --> Route Based on Environment
--> Production DB
--> Test DB
The 'Route Based On Environment' processor could be:
Add a property with name:
production
And value:
${PROD_ENV:matches('TRUE'}
That way out of that processor when you have an environment variable
set called "PROD_ENV" and it is set to "TRUE" the flow will go to
'production' relationship and otherwise it will go to 'no match" which
you can route to test.
This way it is the same flow configuration in both environments but
the path the data will take will vary based on your environment. This
has the same effect as substitution but is more intentional and visual
and it supports more complex differences between prod and test
environments.
Thanks
Joe
On Mon, Feb 23, 2015 at 9:13 AM, Toivo Adams <[email protected]> wrote:
> Corey,
> Thank you for sharing your experience.
>
> Bryan,
> This is often the case.
> Changed and tested custom processor must be tested in test environment using
> flow template.
> And only after successful testing and bureaucracy new nar can be moved to
> live system.
> Template (flow) itself may not be changed at all.
>
> Mark,
> Right, adding Scrambling component to flow is good idea to get automatically
> data for testing and problem solving. I think our security will accept this.
> But mixing test and live environments is strictly prohibited.
> Often this is not even possible, live and test environments are behind
> different firewalls and direct links are not possible.
> So we can get data from live system, but testing must be done in test
> environment.
>
> “If there's one thing that I've learned in my years as a software developer,
> it's that no matter how diligent we are in testing our code, we get data in
> production that we just haven't accounted for.”
> Yes, painfully true and happens too often.
> It's especially embarrassing after spending so many hours for testing. And
> you must explain to business people why well tested application has problems
> in live.
> :)
> But rules are rules and we must follow the rules.
> Over the time when NiFi is accepted hopefully some rules will be changed.
> Before changing any rule reason must be well explained, benefits, risks,
> etc.
>
> Joe,
> Some of our current processes are implemented using Spring framework.
> Spring xml configuration file describes how Java beans are tied together to
> create sort of 'proccesing flow'.
> Of course such configuration is very static and primitive compared to NiFi.
> We keep such configuration files in test environment and separate
> configuration files in live environment. And sometimes they are going out of
> sync. Keeping them is sync is manual work and people do make mistakes.
>
> We have few new applications under development where we have concept of
> Test.conf and Live.conf.
> Application configuration is exactly the same for test and live. Only
> Test.conf and Live.conf contains different information, mostly only
> 'endpoints'. For example DATABASE.
> When application is in test environment it will pick Test.conf
> automatically.
> This is very handy. Test.conf and Live.conf change very rarely.
>
> So now I'll examine the possibilities.
> I'd like always to reduce possibilities to make mistakes and reduce
> development time.
> Remove error-prone manual work, eliminate writing boilerplate code, DRY,
> prefer immutable data, side-effect free code, etc.
>
> “For now is the only thing you're requesting that you'd like to see
> variables that can be substituted? ”
> Yes, nice to have something like that.
> It's not a show-stopper.
> Don't know how this should be implemented.
> One workaround is possibly to create special Controller Service (services)?
> For example https://issues.apache.org/jira/browse/NIFI-322
> New Database Connection Pooling Controller Service
> will offer some functionality.
>
> I have other ideas as well, but let's move step by step.
> :)
>
> Thank for thinking and support
> toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/Great-question-on-nifi-IRC-room-today-NiFi-BPM-sharing-configuration-tp787p811.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at
> Nabble.com.
