Sorry, the one list was meant to be...
a) system properties
b) environment variables
c) process-group variable registry
d) file-based variable registry
On Mon, Oct 19, 2020 at 4:35 PM Bryan Bende <bbe...@gmail.com> wrote:
>
> I think there is some confusion based on terminology...
>
> A given property has an expression language scope defined, which can
> be "flow file attributes" or "variable registry only". This was
> created mostly for documentation purposes so that users could look at
> a property in the docs and see "expression language supported: true"
> and know which values they could reference. What it really comes down
> to in the code is the difference between
> "someProperty.evaluateAttributeExpressions()" and
> "someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
> reference a value from a flow file or not.
>
> In the case of "variable registry only", the value can come from any
> of the following...
> a) system properties
> b) expression language
> c) process-group variable registry
> d) file-based variable registry
>
> When people have stated that "variables are being deprecated in favor
> of parameters", they are referring to the last two items (c & d).
>
> The reason being that parameters solved several short-comings of those
> two options...
>
> - the ability to store sensitive values encrypted
> - the ability to reference them from any property using a new syntax
> #{...}, not long dependent on component developer saying
> "expressionLanguageSupported(true)" on the descriptor
> - the ability to create policies for which users/groups could
> reference a set of parameters
> - the hierarchical ambiguity of what "${foo}" actually resolves to
>
> If you just want to use environment variables, why use parameter
> contexts? Expression language has always offered access to env vars
> and still will going forward.
>
> The one argument I can see is that not all properties support
> expression language, so using parameters gives you a way around that.
>
>
>
>
>
> On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
> <chris.samp...@naimuri.com.invalid> wrote:
> >
> > Being based primarily in Docker containers and having experience with both
> > Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> > environment variables or files) and Docker Swarm (which only handles
> > secrets as environment variables), I'd have definitely been wanting this
> > before moving from Variables to Parameters if I was still in Swarm (or
> > Docker Compose/straight up Docker).
> >
> > It's certainly possible to script creating/updating Parameters via the
> > Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> > possible as a Job in Kubernetes, for example). So environment variables
> > could save the day in that instance.
> >
> > I guess one likely problem (but no different to how I guess the Variable
> > Registry uses env vars) would be how NiFi will handle changes to the env
> > vars - does it:
> >
> > - ignore them until instance restart, which could lead to maintainer
> > confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
> > failing in NiFi)
> > - alert the maintainer to the fact that the env var has changed and a
> > Parameter needs updating, with the new value being used after all
> > associated processors/controllers have been restarted
> > - automatically attempt to update the parameters by restarting all
> > associated processors/controllers, which I'd guess would be a bit
> > dangerous
> > for interrupting in-flow data, etc.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.samp...@naimuri.com
> > <https://www.naimuri.com/>
> >
> >
> > On Mon, 19 Oct 2020 at 19:35, Bryan Bende <bbe...@gmail.com> wrote:
> >
> > > Access to environment variables directly from expression language is
> > > not being removed.
> > >
> > > The discussion is about whether a parameter value should be able to
> > > use expression language to reference an environment variable.
> > >
> > > For example, processor property has #{keystore.password} -> parameter
> > > "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> > > the password from an environment variable.
> > >
> > >
> > >
> > > On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com <u...@moosheimer.com>
> > > wrote:
> > > >
> > > > Chad,
> > > >
> > > > So far I thought that only the NiFi variables are deprecated and access
> > > to environment variables will still be possible.
> > > >
> > > > If this is not the case, then I agree with you. It should definitely be
> > > possible to access environment variables. Otherwise I can't imagine how to
> > > refer to the hostname or the current JAVA path or ... or ... or on each
> > > node?!
> > > >
> > > > Mit freundlichen Grüßen / best regards
> > > > Kay-Uwe Moosheimer
> > > >
> > > > > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky <czobri...@gmail.com>:
> > > > >
> > > > > Andy,
> > > > >
> > > > > Thanks for the response!
> > > > >
> > > > > When I was thinking through this the deprecation of variables was
> > > > > definitely on my mind but the fact that it already had direct access
> > > > > to
> > > > > environment variables was the simplest path. I think it does make more
> > > > > sense to add access to environment variables to the parameter context,
> > > or
> > > > > allowing a specific scope just for environment variables in the
> > > > > expression language.
> > > > >
> > > > > I think giving access to environment variables actually allows more
> > > > > portability between environments, eg dev, test, prod. Defining those
> > > once
> > > > > and allowing for nifi to pull them in makes sense to me and I think is
> > > > > common in container environments.
> > > > >
> > > > > Looking forward to discussing more and better approaches.
> > > > > Chad
> > > > >
> > > > >> On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto <alopre...@apache.org>
> > > wrote:
> > > > >>
> > > > >> Hi Chad,
> > > > >>
> > > > >> Parameters were introduced as a way to deprecate (NiFi) variables
> > > > >> entirely. I’m not sure that introducing a dependency between the two
> > > is a
> > > > >> positive step forward. I think there is a separate conversation to be
> > > had
> > > > >> about allowing parameters access to environment variables, but I
> > > think this
> > > > >> could introduce problems as parameters are designed for flexibility
> > > and
> > > > >> portability, and moving from a system where a parameter was actually
> > > > >> a
> > > > >> pass-through to an environment variable would cause unexpected
> > > problems on
> > > > >> the destination system.
> > > > >>
> > > > >> I think the pros and cons of this need to be clearly enumerated and
> > > > >> discussed here. Thanks for bringing this up.
> > > > >>
> > > > >>
> > > > >> Andy LoPresto
> > > > >> alopre...@apache.org
> > > > >> alopresto.apa...@gmail.com
> > > > >> He/Him
> > > > >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > >>
> > > > >>>> On Oct 19, 2020, at 9:43 AM, Chad Zobrisky <czobri...@gmail.com>
> > > wrote:
> > > > >>>
> > > > >>> Hello,
> > > > >>>
> > > > >>> I was configuring an SSL Context Controller Service today and had
> > > > >>> the
> > > > >>> keystores and passwords passed into the container via environment
> > > > >>> variables. I thought it would be nice to be able to reference these
> > > from
> > > > >>> the parameter context. Maybe either giving Parameter Context values
> > > the
> > > > >>> VARIABLE_REGISTRY scope in the Expression Language, or a new scope
> > > for
> > > > >>> references external to nifi?
> > > > >>>
> > > > >>> I think for refreshing the Parameter Context on those external
> > > changes,
> > > > >> it
> > > > >>> would require an edit/re-apply just as it does now, and would have
> > > > >>> to
> > > > >> make
> > > > >>> sure it is well documented.
> > > > >>>
> > > > >>> I'd be interested in creating a PR for this if the idea makes sense
> > > and
> > > > >> is
> > > > >>> acceptable.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Chad
> > > > >>
> > > > >>
> > > >
> > >
