Joe,
Thanks.
We're not doing it yet, so it's hard to judge whether it's the right
approach or not. We're new at this, so there's some stabbing around in
the dark.
What I'm thinking is that our implementation team want as much at their
fingertips as possible. So they're slaving over a hot NiFi UI all day,
funneling thousands or millions of documents through our ETL. If they
see a flow begin to manifest problems immediately, then they're on it
and want to shut it down immediately. But, they have to see there are
problems. My use case is a) warning that a diskspace threshold is
threatened, b) stop the flow(s) when space dips under a definable threshold.
This is the direction we'd like to go today. I can't predict how we'll
do it better tomorrow, or I'd just do it better right off. Maybe we'll
abandon the disk-monitoring processor. A lot of the innovation right now
is getting used to what problems we have, what we've introduced, what
techniques such as back-pressure are going to solve, trying this, trying
that, etc.
Our implementation team aren't naive guys, but they're seriously under
pressure and don't want to spend too much time looking at cryptic
provenance displays, queue lists, comparing things and doing what the
ETL team, who write the custom processors and dream up flows of those
and standard processors, are willing to do. Implementation just want to
get the job done. They'll tend to say, "Uh, it's broken."
"Yeah, how?"
"Well, it takes too long" or "It filled our disk up" or "We're pretty
sure we lost some of our files", etc.
I'm trying to get them gross-granularity tools whose behavior or data
they can give back to us without them spending so much time doing our
job for us. (Which is not to say that we're not over there looking too,
but what they do and where they do it isn't our "Guys, let's try it
again, but this time..." playground either.)
Hence the awkwardness of some of my questions in this list.
Thanks for listening!
Russ
On 12/02/2016 12:27 PM, Joe Witt wrote:
Russ
Given that our repositories are pluggable it is important that we'd
surface this sort of information via some API friendly mechanism. We
should consider, in my opinion, providing some status gathering
call(s) to the ContentRepository API.
Otherwise, there is not a safe mechanism to do this with. We actually
realized this was being abused previously via the properties mechanism
and refactorings in 1.0 helped make that cleaner so now we need to
just provide a nice mechanism to do what you're seeking.
Having said this, please do also consider just using an external
monitoring service for things like directory usage,etc.. Since you're
only needing this information for passive usage purposes in NiFi there
is probably not much of a benefit of hosting it in nifi.
What do you think?
Thanks
Joe
On Fri, Dec 2, 2016 at 11:11 AM, Russell Bateman <[email protected]> wrote:
/nifi.properties/ locations, such as
nifi.content.repository.directory.default, are typically specified as
relative paths, but I suppose they could be arbitrarily complex. Whatever
the case, I need to discover the actual path to the content subdirectory at
runtime not because I want to create, delete, read or change files there,
but only to know what volume this subdirectory is on so that I can use
File.getUsableSpace()in a custom processor I'm writing that will monitor
space left.
Is there a reliable way to discover this, directly or via a property using
NiFi Expression Language?
Thanks,
Russ
