[
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401310#comment-15401310
]
Joseph Witt commented on NIFI-2208:
-----------------------------------
Above is a PR that reflects the following thoughts.
1)
VariableRegistry should not be editable at least in so far as
extensions/components are concerned so we must ensure the interface doesn't
expose a way to alter state. We'll of course allow it to be editable
eventually but at that point it will be in a different or extended interface
not exposed to components themselves as we'll have to manage the lifecycle of
those edits in a manner that doens't mean a component could have dirty reads
within an execution cycle. So slimmed down the interface to a single method.
Also introduced a new small immutable object class called VariableDescriptor
which is used as the key to interact with elements in the variable registry.
It allows us to capture more than just the String key name such as whether the
variable is sensitive and a description that will help us with logging or user
display cases. Logging will be helpful when people are confused about variable
precedence issues.
2)
Concrete implementations of the VariableRegistry should be avoided in the
nifi-api as much as possible. The nifi-api is our most public and most rigid
definition of extension points so the more we put in there the higher the
friction and the more challenging it is to evolve. Slimmed down to a single
static final instance of a system_environment_variable registry in the
interface of VariableRegistry and then moved the only needed concrete
implementation at this time to nifi-framework-core.
3)
Changes for VariableRegistry had an impact beyond what was probably intended
and beyond what the current lifecycle management really supports. For example
the custom UI for UpdateAttribute had it because it was evaluating expressions.
However, the expression evaluation being done there was purely to validate the
structure of the expression to help with user experience. The actual
expression evaluation is still in the processor where it belongs. But the
VariableRegistry was there seemingly due to constructors requiring it. That
had a ripple effect into the NiFiWebContext which caused an API change for that
and caused spring context properties changes, etc.. Also, the Authorizer API
was updated for variable registry and this should be avoided unless we have a
clear need there. However, in general since we don't have a clear strategy for
the lifecycle managment of the variable registry, in my opinion, we should take
a less is more approach for now. It is also now in nifi-bootstrap though it is
using it for general system/env properties so I kept this in place because it
isn't exposing the user defined aspects of the variable registry we will have.
4)
Due to updates to constructors of various mock contexts and property value
classes it appears the ripple effect was wider than really necessary. Most
often the variable registry isn't needed to test the use case at hand. Added
constructors that made it optional and a sentinel instance of it in the
interface to make null handling more explicit and safe. Was able to remove a
lot of code due to this.
5)
The nifi-api AttributeExpression javadocs needed to be updated to reflect the
intent of the API now that we have the variable registry.
6)
Introduced a ValueLookup class to the expression language module. This class
to help wrap the VariableRegistry with precedence based attributes such as
processor properties and flowfile attributes rather than having the
VariableRegistry support these directly. The VariableRegistry is purely the
user defined properties, system properties, and environment variables so we
should avoid extending it to include other transient/optional items which have
different lifecycles. The ValueLookup class helps encapsulate the difference
between the VariableRegistry and the local needs and interface of expression
language.
7)
To support some of the comments about life cycle concerns we had tests which
altered system properties then grabbed the registry to verify substitution was
working. The problem with this is it goes beyond the intent of system and env
variables being loaded only once at startup. We won't have a chance to get
system properties edited at runtime because if we support that then we run the
risk of variable registry values changing outside our supported lifecycle. So I
removed these tests.
8)
Not a change now but commentary for future: Changes to the api could include
adding support in the PropertyValue class or something else which returns the
referenced VariableDescriptors in a given expression/property. This will allow
us to restrict/manage when users can alter variables in the registry based on
whether components are running that reference them. Further, we now have the
ability to retain whether a variable is sensitive or not. Within the API we
could either decrypt them on demand or we could ensure they are never
represented client-side or in logs.
9)
Updated the logic to reflect that processor properties passed to ValueLookup
get precedence over flowfile attributes as this was the previously documented
behavior and if you think of the principle of most specific/localized values
first this does make sense.
> Support Custom Properties in Expression Language - 1.x baseline
> ---------------------------------------------------------------
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Yolanda M. Davis
> Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can
> close out NIFI-1974 since 0.7.0 is ready to be released.
> In addition to the merge this should also include a fix to ensure that
> variable registry is initialized on startup that variables from the registry
> are applied during EL compilation based on the following order of precedence:
> 1) Flow File Attribute
> 2) Processor provided variables
> 3) User Defined Variables (via custom properties)
> 4) JVM System Properties
> 5) OS Environment Variables
> Finally the following processor properties should be enabled to support
> expression language:
> Put HDFS/Get HDFS/List HDFS
> - Directory property
> ConsumeJMS/PublishJMS
> - Destination Name property
> MS Connection Factory Provider
> -MQ ConnectionFactory Implementation (fqn classname)
> -MQ client library path
> -Broker URI
> DBCP Connection Pool:
> -Database Connection URL
> -Database Driver Class Name
> -DB Driver jar url
> -DB username
> -DB password
> ConvertCSVToAvro
> -add EL support for the following property
> -csv charset
> -and below...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
