Our usage, for ETL, is controlled very up-close and personal right now.
Our ETL of medical documents is pretty involved, changes radically from
customer to customer, and must be baby-sat closely. Anything we're able
to do for our implementation folk to enable them to pin down waste of
resources including and especially time to ingest horrendous quantities
of information is going to serve us for a long time to come. User access
to an easy processor like the one I've written, though what it does is
pretty trivial, makes their life so much easier and they can talk back
to us about where time (in particular) is being spent, in which
processor (we have lots of custom processors that do very
out-of-the-ordinary things), across which subflow, etc.
Except that we anticipate moving to a clustered implementation soon, I
thought about merely looking for a system environment variable or even
the presence of a file, then setting static state inside the processor
to halt doing anything. Conversely, a change to that state might start
the processor back up again (time-stamping, histogramming, etc.). I
think this naïve control strategy falls apart as soon we go to a cluster.
It's taking me a while to get into the NiFi culture, I think. However, I
also think that NiFi folk use NiFi in wildly different ways so maybe how
I'm looking to do something isn't always so un-NiFi, but that others
just haven't tackled it yet.
Yeah, if NiFi gave us some kind of modifiable, global state, especially
if less static than /conf/nifi.properties/, but even if requiring a
bounce to engage it (so, /conf/flow.properties/ or /conf/flow.conf/),
that would solve our problem pretty elegantly. However, I haven't
thought about what problems it also creates for you or others.
Russ
On 11/30/2016 02:42 PM, Joe Witt wrote:
Russ
I don't think we provide anything particularly helpful here to do this
conveniently. You could of course script this external to NiFi to
make HTTP calls to shut off such items. Spitballing ideas here but
what about giving you the ability to tag components with some label
and then be able to do global execution of some task
(stop/start/disable/delete/etc..) against components that you're
authorized to and which have those labels.
Do you think this would be a typical use case or do you feel this is
useful because you're testing right now? Does the above idea make
sense or do you have other suggestions?
Thanks
Joe
On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <[email protected]> wrote:
I've written a custom processor for some trivial profiling, time-stamping,
time-since, histogram-generating, etc., but would like the ability to turn
all instances completely off without having to visit each instance in the
UI. If it works out, I might consider even leaving some instances in
production- or at least staging-environment flows.
1. I know that the NiFi Expression Language has access to various system- or
NiFi properties or settings, but what would someone suggest as best practice
for this? (Don't invade /conf/nifi.properties/, etc.)
2. I guess I'd add a property to configure in my processor and check whether
it evaluates true/false/etc. based on the source data (whatever that will
be--see previous paragraph)?
3. Last, if this processor is thereby reduced merely to
session.transfer( flowfile, SUCCESS );
there isn't any handling even more minimal or faster than that in the sense
of turning a processor off, right?
Thanks for any suggestions,
Russ
