subject:"Static processor design"

Re: Static processor design

2021-01-09 Thread Mark Bean

Russ,

If you're going to process a flowfile by a processor via ProcessSession
(even if "process" does nothing but pass the FF along, unmodified), you
still need to get it from the session and subsequently transfer it to a
relationship. My suggestion was meant to indicate you can handle multiple
flowfiles in a single OnTrigger call thereby moving as many FF's at once.
In other words, only one thread allocation to this processor will move N
flowfiles versus waiting for N allocations of a thread if only moving one
at a time. This will increase throughput and assist in your "as fast as
possible" requirement.

-Mark


On Sat, Jan 9, 2021 at 11:27 AM Russell Bateman 
wrote:

> Mark,
>
> Thanks for responding. I think my question is a little more naive than
> that on my part.
>
> I want to get those files through there as fast as possible. If I ask
> for /n/ files, how many would contribute to them getting them through
> the quickest? After all, I will do nothing at all to any except transfer
> them on and I don't care how many.
>
> I write a lot of custom processors that do specific things to flowfiles
> one at a time. This isn't one of those. I don't care what's coming
> through, I just want to get every flowfile straight through with no
> changes.
>
> Thanks.
>
> Russ
>
> On 1/9/21 9:09 AM, Mark Bean wrote:
> > Russell,
> >
> > You can use "session.get(N)" where N is an integer. This will get up to N
> > flowfiles per OnTrigger() call.
> >
> > -Mark
> >
> >
> > On Fri, Jan 8, 2021 at 5:07 PM Russell Bateman 
> > wrote:
> >
> >> Very well, I have decided to force customer flowfiles through this
> >> processor (I did check out the /Listen/* processors, but chose this
> >> easier solution). This now works. However,
> >>
> >> It brings up another question: is this the most efficient way to pass
> >> flowfiles straight through this processor (we're not processing them in
> >> any way), or is there a batching technique that's faster, etc. I want
> >> this to be straight-through, no back-pressure, throttling or influencing
> >> their passage whatsoever (because I didn't want them coming through in
> >> the first place). It should be ask if this processor weren't there.
> >>
> >> Thanks for any and all thoughts on this.
> >>
> >> public class HumanReadables extends AbstractProcessor{private boolean
> >> propertyModified = false;@Override public void onTrigger( final
> >> ProcessContext context, final ProcessSession session ) throws
> >> ProcessException{FlowFile flowfile = session.get();if( propertyModified
> >> ){propertyModified = false;// record effects of changed
> >> properties...}if( nonNull( flowfile ) )session.transfer( flowfile,
> >> SUCCESS );}...@Override public void onPropertyModified( final
> >> PropertyDescriptor descriptor, final String oldValue, final String
> >> newValue ){propertyModified = true;}
> >>
> >>
>
>

Re: Static processor design

2021-01-09 Thread Russell Bateman


Mark,

Thanks for responding. I think my question is a little more naive than 
that on my part.


I want to get those files through there as fast as possible. If I ask 
for /n/ files, how many would contribute to them getting them through 
the quickest? After all, I will do nothing at all to any except transfer 
them on and I don't care how many.


I write a lot of custom processors that do specific things to flowfiles 
one at a time. This isn't one of those. I don't care what's coming 
through, I just want to get every flowfile straight through with no changes.


Thanks.

Russ

On 1/9/21 9:09 AM, Mark Bean wrote:

Russell,

You can use "session.get(N)" where N is an integer. This will get up to N
flowfiles per OnTrigger() call.

-Mark


On Fri, Jan 8, 2021 at 5:07 PM Russell Bateman 
wrote:


Very well, I have decided to force customer flowfiles through this
processor (I did check out the /Listen/* processors, but chose this
easier solution). This now works. However,

It brings up another question: is this the most efficient way to pass
flowfiles straight through this processor (we're not processing them in
any way), or is there a batching technique that's faster, etc. I want
this to be straight-through, no back-pressure, throttling or influencing
their passage whatsoever (because I didn't want them coming through in
the first place). It should be ask if this processor weren't there.

Thanks for any and all thoughts on this.

public class HumanReadables extends AbstractProcessor{private boolean
propertyModified = false;@Override public void onTrigger( final
ProcessContext context, final ProcessSession session ) throws
ProcessException{FlowFile flowfile = session.get();if( propertyModified
){propertyModified = false;// record effects of changed
properties...}if( nonNull( flowfile ) )session.transfer( flowfile,
SUCCESS );}...@Override public void onPropertyModified( final
PropertyDescriptor descriptor, final String oldValue, final String
newValue ){propertyModified = true;}

Re: Static processor design

2021-01-09 Thread Mark Bean

Russell,

You can use "session.get(N)" where N is an integer. This will get up to N
flowfiles per OnTrigger() call.

-Mark


On Fri, Jan 8, 2021 at 5:07 PM Russell Bateman 
wrote:

> Very well, I have decided to force customer flowfiles through this
> processor (I did check out the /Listen/* processors, but chose this
> easier solution). This now works. However,
>
> It brings up another question: is this the most efficient way to pass
> flowfiles straight through this processor (we're not processing them in
> any way), or is there a batching technique that's faster, etc. I want
> this to be straight-through, no back-pressure, throttling or influencing
> their passage whatsoever (because I didn't want them coming through in
> the first place). It should be ask if this processor weren't there.
>
> Thanks for any and all thoughts on this.
>
> public class HumanReadables extends AbstractProcessor{private boolean
> propertyModified = false;@Override public void onTrigger( final
> ProcessContext context, final ProcessSession session ) throws
> ProcessException{FlowFile flowfile = session.get();if( propertyModified
> ){propertyModified = false;// record effects of changed
> properties...}if( nonNull( flowfile ) )session.transfer( flowfile,
> SUCCESS );}...@Override public void onPropertyModified( final
> PropertyDescriptor descriptor, final String oldValue, final String
> newValue ){propertyModified = true;}
>
>

Re: Static processor design

2021-01-08 Thread Russell Bateman

Very well, I have decided to force customer flowfiles through this 
processor (I did check out the /Listen/* processors, but chose this 
easier solution). This now works. However,


It brings up another question: is this the most efficient way to pass 
flowfiles straight through this processor (we're not processing them in 
any way), or is there a batching technique that's faster, etc. I want 
this to be straight-through, no back-pressure, throttling or influencing 
their passage whatsoever (because I didn't want them coming through in 
the first place). It should be ask if this processor weren't there.


Thanks for any and all thoughts on this.

public class HumanReadables extends AbstractProcessor{private boolean 
propertyModified = false;@Override public void onTrigger( final 
ProcessContext context, final ProcessSession session ) throws 
ProcessException{FlowFile flowfile = session.get();if( propertyModified 
){propertyModified = false;// record effects of changed 
properties...}if( nonNull( flowfile ) )session.transfer( flowfile, 
SUCCESS );}...@Override public void onPropertyModified( final 
PropertyDescriptor descriptor, final String oldValue, final String 
newValue ){propertyModified = true;}

Re: Static processor design

2021-01-08 Thread Otto Fowler

*ListenDynamicPropertyChange

> On Jan 8, 2021, at 12:42, Otto Fowler  wrote:
> 
> So you would be implementing ListDynamicPropertyChange kind of.
> 
> 
>> On Jan 8, 2021, at 12:41, Otto Fowler > > wrote:
>> 
>> What are the attributes of the processor?
>> Do you have : 
>> 
>> @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
>> or anything?
>> 
>> You should take a look at the Listen** processors.
>> They do not have any inputs etc, and run processing queued input from a 
>> background thread.
>> You can have on modified queue a change and then have trigger process the 
>> queue of changes.
>> 
>> 
>> 
>>> On Jan 8, 2021, at 11:04, Russell Bateman >> > wrote:
>>> 
>>> I only put the code I want to execute in onTrigger(), I suspected it would 
>>> not fire there. I know that this isn't what processors do. Configuration is 
>>> a messy problem to solve when your downstreamers want it made easy. This is 
>>> supposed to be a solution that allows them to remain in the NiFi UI and not 
>>> have to run off doing harder things to configure. I could put what I'm 
>>> doing in /HumanReadables/ into a real, running processor, but then, I would 
>>> kind of have to add them to several processors and I wanted to avoid the 
>>> confusion that created.
>>> 
>>> Here's the code. Thanks.
>>> 
>>> import com.windofkeltia.constants.HumanReadableMappings;
>>> 
>>> ...
>>> 
>>> @TriggerWhenEmpty
>>> @SideEffectFree
>>> @CapabilityDescription( "Dynamic properties can be created to specify (or 
>>> add to) static configuration"
>>>   + " of key-value substitution pairs. See additional 
>>> details." )
>>> public class HumanReadables extends AbstractProcessor
>>> {
>>>   @Override
>>>   public void onTrigger( final ProcessContext context, final ProcessSession 
>>> session ) throws ProcessException
>>>   {
>>> getLogger().info( "inside onTrigger()..." );
>>> 
>>> for( Map.Entry< PropertyDescriptor, String > entry : 
>>> context.getProperties().entrySet() )
>>> {
>>>   PropertyDescriptor property = entry.getKey();
>>> 
>>>   // do work here--maybe pre-wipe all key-value pairs if we're just 
>>> going to recreate them?
>>>   final String PROPERTY_NAME  = property.getName();
>>>   final String PROPERTY_VALUE = entry.getValue();
>>> 
>>>   logger.trace( "Processing configurable mappings titled \"" + 
>>> PROPERTY_NAME + "\"" );
>>> 
>>>   try
>>>   {
>>> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>>>   }
>>>   catch( Exception e )
>>>   {
>>> getLogger().debug( e.getMessage() );
>>>   }
>>> }
>>>   }
>>> 
>>>   protected static void harvestDynamicPropertyMappings( final String 
>>> PROPERTY_VALUE )
>>>   {
>>> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
>>> intlineNumber = 0;
>>> 
>>> if( LINES.length < 1 )
>>>   return;
>>> 
>>> final String WHICH_LIST = LINES[ 0 ];
>>> 
>>> for( final String VALUE_LINE : LINES )
>>> {
>>>   char delimiter = VALUE_LINE.charAt( 0 );
>>>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>>>   String key, value;
>>> 
>>>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
>>> VALUE_LINE.substring( 1, position ).trim();
>>>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 
>>> ).trim() : "";
>>> 
>>>   HumanReadableMappings.add( key, value );
>>> }
>>>   }
>>> 
>>>   @Override
>>>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
>>> String propertyDescriptorName )
>>>   {
>>> return new PropertyDescriptor.Builder()
>>>  .required( false )
>>>  .name( propertyDescriptorName )
>>>  .addValidator( 
>>> StandardValidators.NON_EMPTY_VALIDATOR )
>>>  // or .addValidator( Validator.VALID ) if 
>>> you do not wish it validated!
>>>  .dynamic( true )
>>>  .build();
>>>   }
>>> 
>>>   private volatile Set< String > dynamicPropertyNames = new HashSet<>();
>>> 
>>>   @Override
>>>   public void onPropertyModified( final PropertyDescriptor descriptor, 
>>> final String oldValue, final String newValue )
>>>   {
>>> getLogger().info( oldValue + " -> " + newValue );
>>> 
>>> final Set< String > newDynamicPropertyNames = new HashSet<>( 
>>> dynamicPropertyNames );
>>> 
>>> if( isNull( newValue ) )
>>>   newDynamicPropertyNames.remove( descriptor.getName() );
>>> else if( isNull( oldValue ) && descriptor.isDynamic() )
>>>   newDynamicPropertyNames.add( descriptor.getName() );
>>> 
>>> dynamicPropertyNames = Collections.unmodifiableSet( 
>>> newDynamicPropertyNames );
>>> 
>>> final Set< String > allDynamicProperties = dynamicPropertyNames;
>>>   }
>>> 
>>>   @OnScheduled public void

Re: Static processor design

2021-01-08 Thread Otto Fowler

So you would be implementing ListDynamicPropertyChange kind of.


> On Jan 8, 2021, at 12:41, Otto Fowler  wrote:
> 
> What are the attributes of the processor?
> Do you have : 
> 
> @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
> or anything?
> 
> You should take a look at the Listen** processors.
> They do not have any inputs etc, and run processing queued input from a 
> background thread.
> You can have on modified queue a change and then have trigger process the 
> queue of changes.
> 
> 
> 
>> On Jan 8, 2021, at 11:04, Russell Bateman > > wrote:
>> 
>> I only put the code I want to execute in onTrigger(), I suspected it would 
>> not fire there. I know that this isn't what processors do. Configuration is 
>> a messy problem to solve when your downstreamers want it made easy. This is 
>> supposed to be a solution that allows them to remain in the NiFi UI and not 
>> have to run off doing harder things to configure. I could put what I'm doing 
>> in /HumanReadables/ into a real, running processor, but then, I would kind 
>> of have to add them to several processors and I wanted to avoid the 
>> confusion that created.
>> 
>> Here's the code. Thanks.
>> 
>> import com.windofkeltia.constants.HumanReadableMappings;
>> 
>> ...
>> 
>> @TriggerWhenEmpty
>> @SideEffectFree
>> @CapabilityDescription( "Dynamic properties can be created to specify (or 
>> add to) static configuration"
>>   + " of key-value substitution pairs. See additional 
>> details." )
>> public class HumanReadables extends AbstractProcessor
>> {
>>   @Override
>>   public void onTrigger( final ProcessContext context, final ProcessSession 
>> session ) throws ProcessException
>>   {
>> getLogger().info( "inside onTrigger()..." );
>> 
>> for( Map.Entry< PropertyDescriptor, String > entry : 
>> context.getProperties().entrySet() )
>> {
>>   PropertyDescriptor property = entry.getKey();
>> 
>>   // do work here--maybe pre-wipe all key-value pairs if we're just 
>> going to recreate them?
>>   final String PROPERTY_NAME  = property.getName();
>>   final String PROPERTY_VALUE = entry.getValue();
>> 
>>   logger.trace( "Processing configurable mappings titled \"" + 
>> PROPERTY_NAME + "\"" );
>> 
>>   try
>>   {
>> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>>   }
>>   catch( Exception e )
>>   {
>> getLogger().debug( e.getMessage() );
>>   }
>> }
>>   }
>> 
>>   protected static void harvestDynamicPropertyMappings( final String 
>> PROPERTY_VALUE )
>>   {
>> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
>> intlineNumber = 0;
>> 
>> if( LINES.length < 1 )
>>   return;
>> 
>> final String WHICH_LIST = LINES[ 0 ];
>> 
>> for( final String VALUE_LINE : LINES )
>> {
>>   char delimiter = VALUE_LINE.charAt( 0 );
>>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>>   String key, value;
>> 
>>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
>> VALUE_LINE.substring( 1, position ).trim();
>>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 ).trim() 
>> : "";
>> 
>>   HumanReadableMappings.add( key, value );
>> }
>>   }
>> 
>>   @Override
>>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
>> String propertyDescriptorName )
>>   {
>> return new PropertyDescriptor.Builder()
>>  .required( false )
>>  .name( propertyDescriptorName )
>>  .addValidator( 
>> StandardValidators.NON_EMPTY_VALIDATOR )
>>  // or .addValidator( Validator.VALID ) if 
>> you do not wish it validated!
>>  .dynamic( true )
>>  .build();
>>   }
>> 
>>   private volatile Set< String > dynamicPropertyNames = new HashSet<>();
>> 
>>   @Override
>>   public void onPropertyModified( final PropertyDescriptor descriptor, final 
>> String oldValue, final String newValue )
>>   {
>> getLogger().info( oldValue + " -> " + newValue );
>> 
>> final Set< String > newDynamicPropertyNames = new HashSet<>( 
>> dynamicPropertyNames );
>> 
>> if( isNull( newValue ) )
>>   newDynamicPropertyNames.remove( descriptor.getName() );
>> else if( isNull( oldValue ) && descriptor.isDynamic() )
>>   newDynamicPropertyNames.add( descriptor.getName() );
>> 
>> dynamicPropertyNames = Collections.unmodifiableSet( 
>> newDynamicPropertyNames );
>> 
>> final Set< String > allDynamicProperties = dynamicPropertyNames;
>>   }
>> 
>>   @OnScheduled public void processProperties( final ProcessContext context )
>>   {
>> for( Map.Entry< PropertyDescriptor, String > entry : 
>> context.getProperties().entrySet() )
>> {
>>   PropertyDescriptor descriptor = entry.getKey();
>> 
>>   if( descriptor.isDynamic() )
>>

Re: Static processor design

2021-01-08 Thread Otto Fowler

What are the attributes of the processor?
Do you have : 

@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
or anything?

You should take a look at the Listen** processors.
They do not have any inputs etc, and run processing queued input from a 
background thread.
You can have on modified queue a change and then have trigger process the queue 
of changes.



> On Jan 8, 2021, at 11:04, Russell Bateman  wrote:
> 
> I only put the code I want to execute in onTrigger(), I suspected it would 
> not fire there. I know that this isn't what processors do. Configuration is a 
> messy problem to solve when your downstreamers want it made easy. This is 
> supposed to be a solution that allows them to remain in the NiFi UI and not 
> have to run off doing harder things to configure. I could put what I'm doing 
> in /HumanReadables/ into a real, running processor, but then, I would kind of 
> have to add them to several processors and I wanted to avoid the confusion 
> that created.
> 
> Here's the code. Thanks.
> 
> import com.windofkeltia.constants.HumanReadableMappings;
> 
> ...
> 
> @TriggerWhenEmpty
> @SideEffectFree
> @CapabilityDescription( "Dynamic properties can be created to specify (or add 
> to) static configuration"
>   + " of key-value substitution pairs. See additional 
> details." )
> public class HumanReadables extends AbstractProcessor
> {
>   @Override
>   public void onTrigger( final ProcessContext context, final ProcessSession 
> session ) throws ProcessException
>   {
> getLogger().info( "inside onTrigger()..." );
> 
> for( Map.Entry< PropertyDescriptor, String > entry : 
> context.getProperties().entrySet() )
> {
>   PropertyDescriptor property = entry.getKey();
> 
>   // do work here--maybe pre-wipe all key-value pairs if we're just going 
> to recreate them?
>   final String PROPERTY_NAME  = property.getName();
>   final String PROPERTY_VALUE = entry.getValue();
> 
>   logger.trace( "Processing configurable mappings titled \"" + 
> PROPERTY_NAME + "\"" );
> 
>   try
>   {
> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>   }
>   catch( Exception e )
>   {
> getLogger().debug( e.getMessage() );
>   }
> }
>   }
> 
>   protected static void harvestDynamicPropertyMappings( final String 
> PROPERTY_VALUE )
>   {
> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
> intlineNumber = 0;
> 
> if( LINES.length < 1 )
>   return;
> 
> final String WHICH_LIST = LINES[ 0 ];
> 
> for( final String VALUE_LINE : LINES )
> {
>   char delimiter = VALUE_LINE.charAt( 0 );
>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>   String key, value;
> 
>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
> VALUE_LINE.substring( 1, position ).trim();
>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 ).trim() 
> : "";
> 
>   HumanReadableMappings.add( key, value );
> }
>   }
> 
>   @Override
>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
> String propertyDescriptorName )
>   {
> return new PropertyDescriptor.Builder()
>  .required( false )
>  .name( propertyDescriptorName )
>  .addValidator( 
> StandardValidators.NON_EMPTY_VALIDATOR )
>  // or .addValidator( Validator.VALID ) if 
> you do not wish it validated!
>  .dynamic( true )
>  .build();
>   }
> 
>   private volatile Set< String > dynamicPropertyNames = new HashSet<>();
> 
>   @Override
>   public void onPropertyModified( final PropertyDescriptor descriptor, final 
> String oldValue, final String newValue )
>   {
> getLogger().info( oldValue + " -> " + newValue );
> 
> final Set< String > newDynamicPropertyNames = new HashSet<>( 
> dynamicPropertyNames );
> 
> if( isNull( newValue ) )
>   newDynamicPropertyNames.remove( descriptor.getName() );
> else if( isNull( oldValue ) && descriptor.isDynamic() )
>   newDynamicPropertyNames.add( descriptor.getName() );
> 
> dynamicPropertyNames = Collections.unmodifiableSet( 
> newDynamicPropertyNames );
> 
> final Set< String > allDynamicProperties = dynamicPropertyNames;
>   }
> 
>   @OnScheduled public void processProperties( final ProcessContext context )
>   {
> for( Map.Entry< PropertyDescriptor, String > entry : 
> context.getProperties().entrySet() )
> {
>   PropertyDescriptor descriptor = entry.getKey();
> 
>   if( descriptor.isDynamic() )
> getLogger().debug( "Dynamic property named:\n" + 
> descriptor.getName()
> + ", value: " + entry.getValue().replaceAll( 
> "\n", " + " ) );
> }
>   }
> 
>   protected static final String DEFAULT_MAPPING_VALUE = ""
> + "|http://loinc.org |LOINC\n"
> +

Re: Static processor design

2021-01-08 Thread Russell Bateman

I only put the code I want to execute in onTrigger(), I suspected it 
would not fire there. I know that this isn't what processors do. 
Configuration is a messy problem to solve when your downstreamers want 
it made easy. This is supposed to be a solution that allows them to 
remain in the NiFi UI and not have to run off doing harder things to 
configure. I could put what I'm doing in /HumanReadables/ into a real, 
running processor, but then, I would kind of have to add them to several 
processors and I wanted to avoid the confusion that created.


Here's the code. Thanks.

import com.windofkeltia.constants.HumanReadableMappings;

...

@TriggerWhenEmpty
@SideEffectFree
@CapabilityDescription( "Dynamic properties can be created to specify 
(or add to) static configuration"
  + " of key-value substitution pairs. See 
additional details." )

public class HumanReadables extends AbstractProcessor
{
  @Override
  public void onTrigger( final ProcessContext context, final 
ProcessSession session ) throws ProcessException

  {
    getLogger().info( "inside onTrigger()..." );

    for( Map.Entry< PropertyDescriptor, String > entry : 
context.getProperties().entrySet() )

    {
  PropertyDescriptor property = entry.getKey();

  // do work here--maybe pre-wipe all key-value pairs if we're just 
going to recreate them?

  final String PROPERTY_NAME  = property.getName();
  final String PROPERTY_VALUE = entry.getValue();

  logger.trace( "Processing configurable mappings titled \"" + 
PROPERTY_NAME + "\"" );


  try
  {
    harvestDynamicPropertyMappings( PROPERTY_VALUE );
  }
  catch( Exception e )
  {
getLogger().debug( e.getMessage() );
  }
    }
  }

  protected static void harvestDynamicPropertyMappings( final String 
PROPERTY_VALUE )

  {
    final String[] LINES  = PROPERTY_VALUE.split( "\n" );
    int    lineNumber = 0;

    if( LINES.length < 1 )
  return;

    final String WHICH_LIST = LINES[ 0 ];

    for( final String VALUE_LINE : LINES )
    {
  char delimiter = VALUE_LINE.charAt( 0 );
  int  position = VALUE_LINE.indexOf( delimiter, 1 );
  String key, value;

  key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
VALUE_LINE.substring( 1, position ).trim();
  value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 
).trim() : "";


  HumanReadableMappings.add( key, value );
    }
  }

  @Override
  protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( 
final String propertyDescriptorName )

  {
    return new PropertyDescriptor.Builder()
 .required( false )
 .name( propertyDescriptorName )
 .addValidator( 
StandardValidators.NON_EMPTY_VALIDATOR )
 // or .addValidator( Validator.VALID ) 
if you do not wish it validated!

 .dynamic( true )
 .build();
  }

  private volatile Set< String > dynamicPropertyNames = new HashSet<>();

  @Override
  public void onPropertyModified( final PropertyDescriptor descriptor, 
final String oldValue, final String newValue )

  {
    getLogger().info( oldValue + " -> " + newValue );

    final Set< String > newDynamicPropertyNames = new HashSet<>( 
dynamicPropertyNames );


    if( isNull( newValue ) )
  newDynamicPropertyNames.remove( descriptor.getName() );
    else if( isNull( oldValue ) && descriptor.isDynamic() )
  newDynamicPropertyNames.add( descriptor.getName() );

    dynamicPropertyNames = Collections.unmodifiableSet( 
newDynamicPropertyNames );


    final Set< String > allDynamicProperties = dynamicPropertyNames;
  }

  @OnScheduled public void processProperties( final ProcessContext 
context )

  {
    for( Map.Entry< PropertyDescriptor, String > entry : 
context.getProperties().entrySet() )

    {
  PropertyDescriptor descriptor = entry.getKey();

  if( descriptor.isDynamic() )
    getLogger().debug( "Dynamic property named:\n    " + 
descriptor.getName()
    + ", value: " + 
entry.getValue().replaceAll( "\n", " + " ) );

    }
  }

  protected static final String DEFAULT_MAPPING_VALUE = ""
    + "|http://loinc.org |LOINC\n"
    + "|http://snomed.info/sct |SNOMED\n"
    + "|http://www.ama-assn.org/go/cpt  |CPT\n"
    + "|http://aapc.org |CPT\n"
    + "|http://www.nlm.nih.gov/research/umls/rxnorm |RxNorm\n"
    + "|http://hl7.org/fhir/sid/ndc |NDC\n"
    + "|http://hl7.org/fhir/sid/icd-9-cm |ICD-9\n"
    + "|http://hl7.org/fhir/sid/icd-10 |ICD-10\n";

  public static final PropertyDescriptor DEFAULT_MAPPINGS = new 
PropertyDescriptor.Builder()

  .name( "default mappings" )
  .displayName( "Default mappings" )
  .required( false )
  .expressionLanguageSupported( ExpressionLanguageScope.NONE )
  .defaultValue(

Re: Static processor design

2021-01-08 Thread Chris Sampson

Timer Driven/onTrigger (I think) will only fire when an incoming FlowFile
is received, thus triggering the processor to execute.

But you mention not having any connections (presumably incoming or
outgoing), so I guess that's not what you're after?

You could try a Scheduling Strategy of "Cron" and set that to a relatively
low interval I guess? Don't know whether it will work though, I'm not sure
processors were designed to be used in quite this way.

Might be worth your explaining your use case a bit more so the community
can understand and suggest possible solutions/alternatives.

---
*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com



On Fri, 8 Jan 2021 at 15:34, Russell Bateman  wrote:

> The code I really want to run is sitting in onTrigger(), though I could
> move it elsewhere.
>
> Yes, I have tried
>
> *Scheduling Strategy*of Timer driven
> *Run Schedule*of 10 sec
>
> ...but the getLogger().info( "called from onTrigger()" )never reaches
> /logs/nifi-app.log/ (while the logging statement from
> onPropertyModified()does reach the log every time I change properties to
> remove old or introduce new properties).
>
>
> On 1/7/21 6:38 PM, Russell Bateman wrote:
> > (Inadequate title; didn't know what to call it.)
> >
> > I have written a processor that doesn't feature any relationships.
> >
> > It accepts dynamically properties that, in theory, when created (or
> > removed, or values added or changed), and sets data into a class
> > inside my NAR.
> >
> > I wonder, however, at what I expect of it because, while it works in
> > unit testing, it does not in practice. I can sort of guess why, but
> > I'm not sure what to do about it. Given that I can create methods to
> > be called at various opportunities by annotating thus:
> >
> > @OnAdded
> > @OnEnabled
> > @OnRemoved
> > @OnScheduled
> > @OnUnscheduled
> > @OnStopped
> > @OnShutdown
> >
> > There isn't one of these annotations that says to my brain, "When a
> > dynamic property is added, changed or removed, wake up and run this
> > method." Except, of course, for onPropertyModified(). A new property
> > is duly added when created in configuration; my call to
> > getLogger.info()from onPropertyModified()shows
> >
> > 2021-01-07 18:32:51,923 INFO [NiFi Web Server-78]
> > c.windofkeltia.processor.HumanReadables
> > HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null ->
> > |http://hospital.smarthealthit.org|Smart Health IT
> >
> > But, how do I incite some code after the fact to awaken and analyze
> > the newly added configuration then affect the
> > HumanReadableMappingsclass instance?
> >
> > (Hope I haven't explained this too badly. I am willing to attach
> > code--it's a very tiny processor.)
> >
> > Thanks
> >
> >
> >
>
>

Re: Static processor design

2021-01-08 Thread Matt Burgess

Russell,

Are you trying to execute the code in onTrigger() only if a dynamic
property has been modified? If so you could have a boolean member
variable "shouldRun" that gets set to true in onPropertyModified() and
checked/reset in onTrigger(). The scripting components do something
similar, to indicate whether the script needs to be reloaded or not.

Apologies if I'm not understanding correctly, could you share your
code as a Gist or something like that?

Thanks,
Matt

On Fri, Jan 8, 2021 at 10:34 AM Russell Bateman  wrote:
>
> The code I really want to run is sitting in onTrigger(), though I could
> move it elsewhere.
>
> Yes, I have tried
>
> *Scheduling Strategy*of Timer driven
> *Run Schedule*of 10 sec
>
> ...but the getLogger().info( "called from onTrigger()" )never reaches
> /logs/nifi-app.log/ (while the logging statement from
> onPropertyModified()does reach the log every time I change properties to
> remove old or introduce new properties).
>
>
> On 1/7/21 6:38 PM, Russell Bateman wrote:
> > (Inadequate title; didn't know what to call it.)
> >
> > I have written a processor that doesn't feature any relationships.
> >
> > It accepts dynamically properties that, in theory, when created (or
> > removed, or values added or changed), and sets data into a class
> > inside my NAR.
> >
> > I wonder, however, at what I expect of it because, while it works in
> > unit testing, it does not in practice. I can sort of guess why, but
> > I'm not sure what to do about it. Given that I can create methods to
> > be called at various opportunities by annotating thus:
> >
> > @OnAdded
> > @OnEnabled
> > @OnRemoved
> > @OnScheduled
> > @OnUnscheduled
> > @OnStopped
> > @OnShutdown
> >
> > There isn't one of these annotations that says to my brain, "When a
> > dynamic property is added, changed or removed, wake up and run this
> > method." Except, of course, for onPropertyModified(). A new property
> > is duly added when created in configuration; my call to
> > getLogger.info()from onPropertyModified()shows
> >
> > 2021-01-07 18:32:51,923 INFO [NiFi Web Server-78]
> > c.windofkeltia.processor.HumanReadables
> > HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null ->
> > |http://hospital.smarthealthit.org|Smart Health IT
> >
> > But, how do I incite some code after the fact to awaken and analyze
> > the newly added configuration then affect the
> > HumanReadableMappingsclass instance?
> >
> > (Hope I haven't explained this too badly. I am willing to attach
> > code--it's a very tiny processor.)
> >
> > Thanks
> >
> >
> >
>

Re: Static processor design

2021-01-08 Thread Russell Bateman

The code I really want to run is sitting in onTrigger(), though I could 
move it elsewhere.


Yes, I have tried

*Scheduling Strategy*of Timer driven
*Run Schedule*of 10 sec

...but the getLogger().info( "called from onTrigger()" )never reaches 
/logs/nifi-app.log/ (while the logging statement from 
onPropertyModified()does reach the log every time I change properties to 
remove old or introduce new properties).



On 1/7/21 6:38 PM, Russell Bateman wrote:

(Inadequate title; didn't know what to call it.)

I have written a processor that doesn't feature any relationships.

It accepts dynamically properties that, in theory, when created (or 
removed, or values added or changed), and sets data into a class 
inside my NAR.


I wonder, however, at what I expect of it because, while it works in 
unit testing, it does not in practice. I can sort of guess why, but 
I'm not sure what to do about it. Given that I can create methods to 
be called at various opportunities by annotating thus:


@OnAdded
@OnEnabled
@OnRemoved
@OnScheduled
@OnUnscheduled
@OnStopped
@OnShutdown

There isn't one of these annotations that says to my brain, "When a 
dynamic property is added, changed or removed, wake up and run this 
method." Except, of course, for onPropertyModified(). A new property 
is duly added when created in configuration; my call to 
getLogger.info()from onPropertyModified()shows


2021-01-07 18:32:51,923 INFO [NiFi Web Server-78] 
c.windofkeltia.processor.HumanReadables 
HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null -> 
|http://hospital.smarthealthit.org|Smart Health IT


But, how do I incite some code after the fact to awaken and analyze 
the newly added configuration then affect the 
HumanReadableMappingsclass instance?


(Hope I haven't explained this too badly. I am willing to attach 
code--it's a very tiny processor.)


Thanks

Static processor design

2021-01-07 Thread Russell Bateman


(Inadequate title; didn't know what to call it.)

I have written a processor that doesn't feature any relationships.

It accepts dynamically properties that, in theory, when created (or 
removed, or values added or changed), and sets data into a class inside 
my NAR.


I wonder, however, at what I expect of it because, while it works in 
unit testing, it does not in practice. I can sort of guess why, but I'm 
not sure what to do about it. Given that I can create methods to be 
called at various opportunities by annotating thus:


   @OnAdded
   @OnEnabled
   @OnRemoved
   @OnScheduled
   @OnUnscheduled
   @OnStopped
   @OnShutdown

There isn't one of these annotations that says to my brain, "When a 
dynamic property is added, changed or removed, wake up and run this 
method." Except, of course, for onPropertyModified(). A new property is 
duly added when created in configuration; my call to 
getLogger.info()from onPropertyModified()shows


2021-01-07 18:32:51,923 INFO [NiFi Web Server-78] 
c.windofkeltia.processor.HumanReadables 
HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null -> 
|http://hospital.smarthealthit.org|Smart Health IT


But, how do I incite some code after the fact to awaken and analyze the 
newly added configuration then affect the HumanReadableMappingsclass 
instance?


(Hope I haven't explained this too badly. I am willing to attach 
code--it's a very tiny processor.)


Thanks

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Re: Static processor design

Static processor design

12 matches

Site Navigation

Mail list logo

Footer information