Re: Get all available variables in the InvokeScriptedProcessor

[Saloni Udani]

Thanks Matt.
Seems like what I am trying to achieve is an antipattern.

On Wed, Aug 12, 2020 at 12:59 AM Matt Burgess  wrote:

> Although this is an "unnatural" use of Groovy (and a conversation much
> better suited for the dev list :), it is possible to get at a map of
> defined variables (key and value). This counts on particular
> implementations of the API and that there is no SecurityManager
> installed in the JVM so Groovy ignores boundaries like private
> classes.  In InvokeScriptedProcessor or ExecuteScript it would look
> something like:
>
> def varRegistry = context.procNode.variableRegistry
> def varMap = [:] as Map
> storeVariables(varMap, varRegistry)
>
> The storeVariables method is just a parent-first recursive call to
> fill your map with variables, this allows child registries to override
> variables that were declared "above":
>
> def storeVariables(map, registry) {
>   if(!registry) return map
>   def parent
> try {
> parent = registry.parent
> } catch(t) {
> map.putAll(registry.variableMap)
>   return map
> }
>   if(!parent) {
> map.putAll(registry.variableMap)
> return map
>   }
>   storeVariables(map, parent)
> }
>
> It works because "context" happens to be a StandardProcessContext
> instance, which has a private "procNode" member of type ProcessorNode,
> which is an extension of AbstractComponentNode which has a
> getVariableRegistry() method.
>
> It's definitely a hack so please use at your own risk :)
>
> Regards,
> Matt
>
> On Tue, Aug 11, 2020 at 1:18 AM Saloni Udani 
> wrote:
> >
> > Thanks Andy, but with expression language I can only get values of
> attributes and not both key and value. In our case , variable key also has
> some useful information.
> >
> > Thanks
> >
> > On Mon, Aug 10, 2020 at 10:32 PM Andy LoPresto 
> wrote:
> >>
> >> Those variables are available to be referenced via Expression Language
> in the flowfile attributes. They are not intended for direct programmatic
> access via code, so you don’t need to address them directly in your Groovy
> code.
> >>
> >> If you need to populate specific values at configuration time, you can
> define dynamic properties on the processor config and reference those
> directly in code (see any existing processor source for examples).
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> alopresto.apa...@gmail.com
> >> He/Him
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Aug 9, 2020, at 10:40 PM, Saloni Udani 
> wrote:
> >>
> >> Thanks Andy.
> >>
> >> By variables I meant NiFi process group variables.
> >> 
> >>
> >> On Sat, Aug 8, 2020 at 12:39 AM Andy LoPresto 
> wrote:
> >>>
> >>> I think we need additional clarification on what you mean by
> “variables”. If you are referring to actual Groovy variables, you can
> enumerate them using the binding available in the context (see below). If
> you mean the attributes available on a flowfile, you can access them
> similarly.
> >>>
> >>> Find all variables starting with prefix:
> >>>
> >>> def varsStartingWithABC = this.binding.variables.findAll { k,v ->
> k.startsWith(“a.b.c”) }
> >>>
> >>> Find all attributes starting with prefix:
> >>>
> >>> def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v ->
> k.startsWith(“a.b.c”) }
> >>>
> >>>
> >>>
> >>> Andy LoPresto
> >>> alopre...@apache.org
> >>> alopresto.apa...@gmail.com
> >>> He/Him
> >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>>
> >>> On Aug 7, 2020, at 2:11 AM, Saloni Udani 
> wrote:
> >>>
> >>> Hi,
> >>> We use NiFi 1.5.0.
> >>> Our use case is to get particular key pattern variables (key and
> value) in the groovy InvokeScriptedProcessor. E.g I want all variables
> whose key starts with "a.b.c". By this I can write a generic logic on
> certain categories of variables for further use.
> >>>
> >>> Is there a way programmatically to get all variables with a certain
> key pattern? Or for that matter is there a way programmatically to get all
> available variables Map ? In NiFi 1.5.0 or further versions.
> >>>
> >>>
> >>> Thanks
> >>> Saloni Udani
> >>>
> >>>
> >>
>

Re: Get all available variables in the InvokeScriptedProcessor

[Matt Burgess]

Although this is an "unnatural" use of Groovy (and a conversation much
better suited for the dev list :), it is possible to get at a map of
defined variables (key and value). This counts on particular
implementations of the API and that there is no SecurityManager
installed in the JVM so Groovy ignores boundaries like private
classes.  In InvokeScriptedProcessor or ExecuteScript it would look
something like:

def varRegistry = context.procNode.variableRegistry
def varMap = [:] as Map
storeVariables(varMap, varRegistry)

The storeVariables method is just a parent-first recursive call to
fill your map with variables, this allows child registries to override
variables that were declared "above":

def storeVariables(map, registry) {
  if(!registry) return map
  def parent
try {
parent = registry.parent
} catch(t) {
map.putAll(registry.variableMap)
  return map
}
  if(!parent) {
map.putAll(registry.variableMap)
return map
  }
  storeVariables(map, parent)
}

It works because "context" happens to be a StandardProcessContext
instance, which has a private "procNode" member of type ProcessorNode,
which is an extension of AbstractComponentNode which has a
getVariableRegistry() method.

It's definitely a hack so please use at your own risk :)

Regards,
Matt

On Tue, Aug 11, 2020 at 1:18 AM Saloni Udani  wrote:
>
> Thanks Andy, but with expression language I can only get values of  
> attributes and not both key and value. In our case , variable key also has 
> some useful information.
>
> Thanks
>
> On Mon, Aug 10, 2020 at 10:32 PM Andy LoPresto  wrote:
>>
>> Those variables are available to be referenced via Expression Language in 
>> the flowfile attributes. They are not intended for direct programmatic 
>> access via code, so you don’t need to address them directly in your Groovy 
>> code.
>>
>> If you need to populate specific values at configuration time, you can 
>> define dynamic properties on the processor config and reference those 
>> directly in code (see any existing processor source for examples).
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Aug 9, 2020, at 10:40 PM, Saloni Udani  wrote:
>>
>> Thanks Andy.
>>
>> By variables I meant NiFi process group variables.
>> 
>>
>> On Sat, Aug 8, 2020 at 12:39 AM Andy LoPresto  wrote:
>>>
>>> I think we need additional clarification on what you mean by “variables”. 
>>> If you are referring to actual Groovy variables, you can enumerate them 
>>> using the binding available in the context (see below). If you mean the 
>>> attributes available on a flowfile, you can access them similarly.
>>>
>>> Find all variables starting with prefix:
>>>
>>> def varsStartingWithABC = this.binding.variables.findAll { k,v -> 
>>> k.startsWith(“a.b.c”) }
>>>
>>> Find all attributes starting with prefix:
>>>
>>> def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v -> 
>>> k.startsWith(“a.b.c”) }
>>>
>>>
>>>
>>> Andy LoPresto
>>> alopre...@apache.org
>>> alopresto.apa...@gmail.com
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Aug 7, 2020, at 2:11 AM, Saloni Udani  wrote:
>>>
>>> Hi,
>>> We use NiFi 1.5.0.
>>> Our use case is to get particular key pattern variables (key and value) in 
>>> the groovy InvokeScriptedProcessor. E.g I want all variables whose key 
>>> starts with "a.b.c". By this I can write a generic logic on certain 
>>> categories of variables for further use.
>>>
>>> Is there a way programmatically to get all variables with a certain key 
>>> pattern? Or for that matter is there a way programmatically to get all 
>>> available variables Map ? In NiFi 1.5.0 or further versions.
>>>
>>>
>>> Thanks
>>> Saloni Udani
>>>
>>>
>>

Re: Get all available variables in the InvokeScriptedProcessor

[Saloni Udani]

Thanks Andy, but with expression language I can only get values of
attributes and not both key and value. In our case , variable key also has
some useful information.

Thanks

On Mon, Aug 10, 2020 at 10:32 PM Andy LoPresto  wrote:

> Those variables are available to be referenced via Expression Language in
> the flowfile attributes. They are not intended for direct programmatic
> access via code, so you don’t need to address them directly in your Groovy
> code.
>
> If you need to populate specific values at configuration time, you can
> define dynamic properties on the processor config and reference those
> directly in code (see any existing processor source for examples).
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 9, 2020, at 10:40 PM, Saloni Udani 
> wrote:
>
> Thanks Andy.
>
> By variables I meant NiFi process group variables.
> 
>
> On Sat, Aug 8, 2020 at 12:39 AM Andy LoPresto 
> wrote:
>
>> I think we need additional clarification on what you mean by “variables”.
>> If you are referring to actual Groovy variables, you can enumerate them
>> using the binding available in the context (see below). If you mean the
>> attributes available on a flowfile, you can access them similarly.
>>
>> Find all variables starting with prefix:
>>
>> def varsStartingWithABC = this.binding.variables.findAll { k,v ->
>> k.startsWith(“a.b.c”) }
>>
>> Find all attributes starting with prefix:
>>
>> def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v ->
>> k.startsWith(“a.b.c”) }
>>
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Aug 7, 2020, at 2:11 AM, Saloni Udani 
>> wrote:
>>
>> Hi,
>> We use NiFi 1.5.0.
>> Our use case is to get particular key pattern variables (key and value)
>> in the groovy InvokeScriptedProcessor. E.g I want all variables whose key
>> starts with "a.b.c". By this I can write a generic logic on certain
>> categories of variables for further use.
>>
>> Is there a way programmatically to get all variables with a certain key
>> pattern? Or for that matter is there a way programmatically to get all
>> available variables Map ? In NiFi 1.5.0 or further versions.
>>
>>
>> Thanks
>> Saloni Udani
>>
>>
>>
>

Re: Get all available variables in the InvokeScriptedProcessor

[Andy LoPresto]

Those variables are available to be referenced via Expression Language in the 
flowfile attributes. They are not intended for direct programmatic access via 
code, so you don’t need to address them directly in your Groovy code. 

If you need to populate specific values at configuration time, you can define 
dynamic properties on the processor config and reference those directly in code 
(see any existing processor source for examples).  

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 9, 2020, at 10:40 PM, Saloni Udani  wrote:
> 
> Thanks Andy.
> 
> By variables I meant NiFi process group variables.
> 
> 
> On Sat, Aug 8, 2020 at 12:39 AM Andy LoPresto  > wrote:
> I think we need additional clarification on what you mean by “variables”. If 
> you are referring to actual Groovy variables, you can enumerate them using 
> the binding available in the context (see below). If you mean the attributes 
> available on a flowfile, you can access them similarly. 
> 
> Find all variables starting with prefix: 
> 
> def varsStartingWithABC = this.binding.variables.findAll { k,v -> 
> k.startsWith(“a.b.c”) }
> 
> Find all attributes starting with prefix:
> 
> def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v -> 
> k.startsWith(“a.b.c”) }
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Aug 7, 2020, at 2:11 AM, Saloni Udani > > wrote:
>> 
>> Hi,
>> We use NiFi 1.5.0.
>> Our use case is to get particular key pattern variables (key and value) in 
>> the groovy InvokeScriptedProcessor. E.g I want all variables whose key 
>> starts with "a.b.c". By this I can write a generic logic on certain 
>> categories of variables for further use.
>> 
>> Is there a way programmatically to get all variables with a certain key 
>> pattern? Or for that matter is there a way programmatically to get all 
>> available variables Map ? In NiFi 1.5.0 or further versions.
>> 
>> 
>> Thanks
>> Saloni Udani
> 

Re: Get all available variables in the InvokeScriptedProcessor

[Andy LoPresto]

I think we need additional clarification on what you mean by “variables”. If 
you are referring to actual Groovy variables, you can enumerate them using the 
binding available in the context (see below). If you mean the attributes 
available on a flowfile, you can access them similarly. 

Find all variables starting with prefix: 

def varsStartingWithABC = this.binding.variables.findAll { k,v -> 
k.startsWith(“a.b.c”) }

Find all attributes starting with prefix:

def attrsStartingWithABC = flowfile.getAttributes().findAll { k,v -> 
k.startsWith(“a.b.c”) }



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 7, 2020, at 2:11 AM, Saloni Udani  wrote:
> 
> Hi,
> We use NiFi 1.5.0.
> Our use case is to get particular key pattern variables (key and value) in 
> the groovy InvokeScriptedProcessor. E.g I want all variables whose key starts 
> with "a.b.c". By this I can write a generic logic on certain categories of 
> variables for further use.
> 
> Is there a way programmatically to get all variables with a certain key 
> pattern? Or for that matter is there a way programmatically to get all 
> available variables Map ? In NiFi 1.5.0 or further versions.
> 
> 
> Thanks
> Saloni Udani

Get all available variables in the InvokeScriptedProcessor

[Saloni Udani]

Hi,
We use NiFi 1.5.0.
Our use case is to get particular key pattern variables (key and value) in
the groovy InvokeScriptedProcessor. E.g I want all variables whose key
starts with "a.b.c". By this I can write a generic logic on certain
categories of variables for further use.

Is there a way programmatically to get all variables with a certain key
pattern? Or for that matter is there a way programmatically to get all
available variables Map ? In NiFi 1.5.0 or further versions.


Thanks
Saloni Udani