Hello Tim,

If I'd understand your requirement correctly, I'd first try using
following flow using the existing processors:

FetchDistributedMapCache
-> RouteOnAttribute
  --  permit --> InvokeHTTP
  --  unmatched --> WhateverAlternativeRoute

Assuming input FlowFiles have a timestamp denoting when the request is
initiated.
First, FetchDistributeMapCache or other FetchXXXX processors can be
used to retrieve available time window for making requests against the
target HTTP service.
E.g. by a epoch long timestamp to represent end timestamp.
Then RouteOnAttribute can be used to route input FlowFiles by EL
something like ${requestTimestamp:lt(${permitRequestUntil})} to the
'permit' relationship.

This way, you can control service availability outside of this part.
By updating the time schedule using a corresponding processor, e.g.
PutDistributedMapCache.

The concern for FlowFiles those passed RouteOnAttribute, but being
queued in front of InvokeHTTP in above example, how to limit such
FlowFile to enter such state?
I would configure the 'permit' connection Back Pressure Object
Threshold as 1 (or grater but keep low based on InvokeHTTP
throughput).
By doing so, RouteOnAttribute will not be scheduled until the queued
FlowFile gets processed, thus no more FlowFiles can proceed.

Hope this helps,
Koji


On Fri, Apr 13, 2018 at 1:20 PM, Tim Dean <tim.d...@gmail.com> wrote:
> Hello,
>
> I have a custom NiFi processor that invokes an external HTTP endpoint. That
> endpoint will be hosted by services running at customer sites, and those
> customer sites require the ability to define when the service can be called
> by my processor. Their goal is to prevent calls from coming in during their
> peak hours so that they only have to process my requests during a
> configurable set of off-peak hours.
>
> Additionally, we have a goal of keeping the code for making the HTTP request
> separate from the code for checking whether or not it is currently in a time
> window that requests are allowed. This is not a strict requirement, but we
> have many different scenarios that would like to use the HTTP request
> processor without any schedule restrictions and still other scenarios that
> would like to check schedule restrictions before running other processors.
>
> My first idea for this was to implement 2 different custom processors, one
> to make the HTTP request and another to check the current time against the
> configured schedule restrictions. Flow files would first enter the schedule
> restriction processor, and transfer to a “success” relationship only if the
> request is currently permitted against the schedule. That success
> relationship would then be connected to the HTTP request processor.
>
> The potential problem I see with this is that flow files could back up for
> some reason between the schedule restriction check processor and the HTTP
> requests. So a flow file could pass the schedule restriction check, wait for
> a while until the HTTP request processor picks up the work, and then end up
> sending an HTTP request outside of the permitted schedule window.
>
> I could avoid this problem completely by combining the logic into a single
> processor, but that makes it more difficult to reuse these processors in
> different combinations for the other scenarios mentioned above.
>
> I’m looking for other options to consider for addressing this workflow. I
> have a couple of thoughts:
>
> Implement the HTTP processor independently and then a second processor that
> subclasses the first to add schedule restrictions. This keeps the two bits
> of code separate but doesn’t give as much flexibility as I’d like
> Just implement this as 2 separate processors and try to figure out some way
> in the flow to prevent flow files from backing up between these 2 processors
> (not sure if this is possible)
> Implement the schedule restriction as a particular implementation of a
> controller service interface, and have the HTTP request processor depend on
> an instance of that controller. Alternate implementations of that controller
> service interface could be created that exclude the schedule restriction
> check.
>
>
> Any thoughts on these approaches? Do any alternatives come to mind that I am
> missing?
>
> Thanks in advance
>
> -Tim

Reply via email to