Re: Unable to catch ProcessException error in JUnit test code

[Russell Bateman]

Yes, but as I never regain control after throwing ProcessorErrorin 
onTrigger(), I don't get a chance to look at the resulting log in my 
unit test. I must still be missing something here?



On 08/25/2016 11:29 AM, Matt Burgess wrote:

Russ,

You can call runner.getLogger() which gives you a MockComponentLog:
https://github.com/apache/nifi/blob/master/nifi-mock/src/main/java/org/apache/nifi/util/MockComponentLog.java.
 From that you can get your messages and check if you logged the
specific one(s) you care about.

Regards,
Matt

On Thu, Aug 25, 2016 at 1:22 PM, Russell Bateman
<russell.bate...@perfectsearchcorp.com> wrote:

Matt,

Thank you for replying. It extends AbstractProcessor:

public class X12MessageRouter extends AbstractProcessor
{
   @Override
   public void onTrigger( final ProcessContext context, final ProcessSession
session ) throws ProcessException
   {
 FlowFile flowfile = session.get();
 ...

No, you're right, I don't want to propagate exceptions outside the
processor. I wasn't even thinking about that. I really just wanted to test
that I'm throwing ProcessorException with the specific message for the
specific reason I'm throwing it. Do you have a suggestion based on this?

Thanks,

Russ


On 08/25/2016 11:12 AM, Matt Burgess wrote:

Does your processor extend AbstractProcessor or
AbstractSessionFactoryProcessor? If the former, then all throwables
get caught by the onTrigger method:

https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java

Are you sure you want such exceptions to propagate out of the
processor? Usually you would route a flow file to some failure or
retry relationship, or rollback the session (which is what
AbstractProcessor does). This keeps the integrity of the framework
rather than an exception escaping into the wild.

Regards,
Matt

On Thu, Aug 25, 2016 at 12:51 PM, Russell Bateman
<russell.bate...@perfectsearchcorp.com> wrote:

  From my custom processor, I'm throwing a ProcessException that never
reaches
my catch in the JUnit test. I already have several other JUnit tests
working
on this processor; this is the remaining wiggle I want to test.

I might be blind here, but I can't see what's wrong. Any comment will be
useful.

Thanks,

Russ

Here's the tail-end of my processor'sonTrigger() method:

  ...

  try
  {
X12Simple x12 = ( X12Simple ) parser.parse( is.get() );
...

session.transfer( newFlowfile, resultingRelationship );
  }
  catch( FormatException e ) *// this exception is caught--was thrown
in
X12 parser as expected!*
  {
session.remove( newFlowfile );
*  throw new ProcessException( e + " (the content is unparsable as an
X12 message)" );*
  }
  catch( IOException e )
  {
session.remove( newFlowfile );
throw new ProcessException( e );
  }
}
finally
{
  session.transfer( flowfile, ORIGINAL );
}
}


And here's the JUnit test case for now:

@Test
public void testOnTriggerCompleteCrap()
{
TestRunner runner = TestRunners.newTestRunner( new X12MessageRouter()
);

final int ONE = 1;
final InputStream MESSAGE = new ByteArrayInputStream( _CRAP.getBytes()
);

Map< String, String > flowFileAttributes = new HashMap<>();
flowFileAttributes.put( "testname", "X12 Message Router Unit Test" );

runner.setValidateExpressionUsage( false );
runner.enqueue( MESSAGE, flowFileAttributes );

try
{
*runner.run( ONE ); **// (exception will happen under here)*
  runner.assertQueueEmpty();
  List< MockFlowFile > results = runner.getFlowFilesForRelationship(
X12MessageRouter.NONE );
}
catch( ProcessException e )
{
  int x = 9; *// (never caught at breakpoint set here)*
}
}

Re: Unable to catch ProcessException error in JUnit test code

[Russell Bateman]

Matt,

Thank you for replying. It extends AbstractProcessor:

public class X12MessageRouter extends AbstractProcessor
{
  @Override
  public void onTrigger( final ProcessContext context, final 
ProcessSession session ) throws ProcessException

  {
FlowFile flowfile = session.get();
...

No, you're right, I don't want to propagate exceptions outside the 
processor. I wasn't even thinking about that. I really just wanted to 
test that I'm throwing ProcessorException with the specific message for 
the specific reason I'm throwing it. Do you have a suggestion based on this?


Thanks,

Russ

On 08/25/2016 11:12 AM, Matt Burgess wrote:

Does your processor extend AbstractProcessor or
AbstractSessionFactoryProcessor? If the former, then all throwables
get caught by the onTrigger method:
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java

Are you sure you want such exceptions to propagate out of the
processor? Usually you would route a flow file to some failure or
retry relationship, or rollback the session (which is what
AbstractProcessor does). This keeps the integrity of the framework
rather than an exception escaping into the wild.

Regards,
Matt

On Thu, Aug 25, 2016 at 12:51 PM, Russell Bateman
<russell.bate...@perfectsearchcorp.com> wrote:

 From my custom processor, I'm throwing a ProcessException that never reaches
my catch in the JUnit test. I already have several other JUnit tests working
on this processor; this is the remaining wiggle I want to test.

I might be blind here, but I can't see what's wrong. Any comment will be
useful.

Thanks,

Russ

Here's the tail-end of my processor'sonTrigger() method:

 ...

 try
 {
   X12Simple x12 = ( X12Simple ) parser.parse( is.get() );
   ...

   session.transfer( newFlowfile, resultingRelationship );
 }
 catch( FormatException e ) *// this exception is caught--was thrown in
X12 parser as expected!*
 {
   session.remove( newFlowfile );
*  throw new ProcessException( e + " (the content is unparsable as an
X12 message)" );*
 }
 catch( IOException e )
 {
   session.remove( newFlowfile );
   throw new ProcessException( e );
 }
   }
   finally
   {
 session.transfer( flowfile, ORIGINAL );
   }
}


And here's the JUnit test case for now:

@Test
public void testOnTriggerCompleteCrap()
{
   TestRunner runner = TestRunners.newTestRunner( new X12MessageRouter() );

   final int ONE = 1;
   final InputStream MESSAGE = new ByteArrayInputStream( _CRAP.getBytes() );

   Map< String, String > flowFileAttributes = new HashMap<>();
   flowFileAttributes.put( "testname", "X12 Message Router Unit Test" );

   runner.setValidateExpressionUsage( false );
   runner.enqueue( MESSAGE, flowFileAttributes );

   try
   {
*runner.run( ONE ); **// (exception will happen under here)*
 runner.assertQueueEmpty();
 List< MockFlowFile > results = runner.getFlowFilesForRelationship(
X12MessageRouter.NONE );
   }
   catch( ProcessException e )
   {
 int x = 9; *// (never caught at breakpoint set here)*
   }
}

Re: 1.0 Run Status Colors

[Russell Bateman]

+1

I like your changes, Peter!

On 08/23/2016 03:12 PM, Peter Wicks (pwicks) wrote:

I have the before/after Status Color images made up and listed in the notes on 
the JIRA: https://issues.apache.org/jira/browse/NIFI-2603

The screenshots are perfect, as there is some tweaking left to be done, but it 
should give a clear picture of the effect the change would have.
One issue I haven't worked out is that in some of the screenshots the "Issues" 
icon is the same orange color it is in the 1.0.0 Beta.  It is supposed to be the color 
shown in the Process Group/Process before/after images.

-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com]
Sent: Friday, August 19, 2016 9:02 AM
To: dev@nifi.apache.org
Subject: Re: 1.0 Run Status Colors

The discussion around the modernized UI goes back to start of the year.

   https://issues.apache.org/jira/browse/NIFI-1323
   https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface

UI changes are notoriously exciting and horrible and for each person they might 
feel neither, one or both of these.

Peter: Thank you for providing the screenshot.  To help better understand your 
suggestion can you show a before, after, and provide some text about the 
changes you'd like?

Your screenshot looks great to me but it also isn't obvious what changed.

On Fri, Aug 19, 2016 at 7:49 AM, Russell Bateman 
<russell.bate...@perfectsearchcorp.com> wrote:

As a consumer of NiFi and a developer of proprietary processors I'm a
little concerned by what I see here too. In fact, I'd like to read
some document supporting/justifying the UI changes, which is not to
say that I presume to condemn them, just that I'd like to understand the 
reasoning.
The existing NiFi UI was already very good.

NiFi rocks; I hope it will keep rocking!

On Thu, Aug 18, 2016 at 8:37 PM, Peter Wicks (pwicks)
<pwi...@micron.com>
wrote:


I created a ticket and put in a link to a mostly accurate screenshot,
at least close enough to give you a feel for what I was thinking:
https://issues.apache.org/jira/browse/NIFI-2603

Direct screenshot link: https://goo.gl/photos/RZzz45RGeTAmWwiC7


-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com]
Sent: Wednesday, August 17, 2016 3:46 PM
To: dev@nifi.apache.org
Subject: Re: 1.0 Run Status Colors

no big discussion really - just contributions.  If you could kick off
a JIRA and attach a screenshot of what you have in mind that could
help the discussion move along.

On Wed, Aug 17, 2016 at 2:33 PM, Peter Wicks (pwicks)
<pwi...@micron.com>
wrote:

We've been playing with NiFi 1.0 and of course the new theme/layout
is a

big change.

One thing we don't particularly care for is the bland gray

Running/Stopped/Disabled colors. I would be willing to have them
bland in the header, but on the Processor/Process Group level I feel
they really need to be Green/Red/whatever color.

I would be happy to submit the change, but I wanted to make sure
there

hadn't been some kind heated discussion over colors or lack thereof
before putting in a PR.

Re: 1.0 Run Status Colors

[Russell Bateman]

As a consumer of NiFi and a developer of proprietary processors I'm a
little concerned by what I see here too. In fact, I'd like to read some
document supporting/justifying the UI changes, which is not to say that I
presume to condemn them, just that I'd like to understand the reasoning.
The existing NiFi UI was already very good.

NiFi rocks; I hope it will keep rocking!

On Thu, Aug 18, 2016 at 8:37 PM, Peter Wicks (pwicks) 
wrote:

> I created a ticket and put in a link to a mostly accurate screenshot, at
> least close enough to give you a feel for what I was thinking:
> https://issues.apache.org/jira/browse/NIFI-2603
>
> Direct screenshot link: https://goo.gl/photos/RZzz45RGeTAmWwiC7
>
>
> -Original Message-
> From: Joe Witt [mailto:joe.w...@gmail.com]
> Sent: Wednesday, August 17, 2016 3:46 PM
> To: dev@nifi.apache.org
> Subject: Re: 1.0 Run Status Colors
>
> no big discussion really - just contributions.  If you could kick off a
> JIRA and attach a screenshot of what you have in mind that could help the
> discussion move along.
>
> On Wed, Aug 17, 2016 at 2:33 PM, Peter Wicks (pwicks) 
> wrote:
> > We've been playing with NiFi 1.0 and of course the new theme/layout is a
> big change.
> > One thing we don't particularly care for is the bland gray
> Running/Stopped/Disabled colors. I would be willing to have them bland in
> the header, but on the Processor/Process Group level I feel they really
> need to be Green/Red/whatever color.
> >
> > I would be happy to submit the change, but I wanted to make sure there
> hadn't been some kind heated discussion over colors or lack thereof before
> putting in a PR.
>

Re: A few NiFi ReST questions...

[Russell Bateman]

Yup, this works very well:

http://localhost:8080/nifi-api/controller/process-groups/*root*/status

Thanks,

Russ

On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example”  on the canvas,
> I get this response:
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> 
> 
> 392b161b-d9cc-442a-9394-8db9cee33ba7
> 
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> 0 / 0 bytes
> NiFi Flow
> 0 / 0 bytes
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> b642a34d-c18c-4bb1-b836-6bbc9d32f31e
> 0 / 0 bytes
> *Token Example*
> 0 / 0 bytes
> 0 bytes
> Invalid
> 0
> 00:00:00.000
> *Base64EncodeContent*
> 0 bytes
> 
> 0 / 0 bytes
> 0
> 0 bytes
> 0 bytes
> 0 / 0 bytes
> 0 / 0 bytes
> 16:09:46 PDT
> 0 / 0 bytes
> 0 bytes
> 
> 
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <matt.c.gil...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
>  1. How many times a processor processed anything?
>  2. How many flowfiles were processed?
>  3. How many flowfiles were produced?
>  4. Can I get this information per processor?
>  5. How can I tell different instances of the same processor apart?
>  6. Why can't I see the name I gave a processor in configuration? (For
>  example, I named an instance of GetFile to "Get PDF files from test
> fodder".
>  7. How can I get a list of items on the NiFi canvas like processors and
>  relationships.
>  8. How many times processors or a processor failed?
>  9. What is the profile of resource usage, like memory in use?
>  10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>
>

Re: A few NiFi ReST questions...

[Russell Bateman]

Andy,

Thanks. Yes, I know how Ajax works having used it before. It's looking at
it happening in the browser developer tools that I've never learned to do.
I've been using  RESTED so that I can set Accept application/json which
works perfectly.

Russ


On Fri, Jul 8, 2016 at 10:55 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Russell,
>
> No worries. The Javascript application that runs the UI is simply invoking
> Ajax (Asynchronous Javascript and XML) requests to normal HTTP endpoints
> provided by the embedded Jetty server in NiFi. You don’t need to understand
> anything about front-end development to monitor these requests using the
> Dev Tools. In fact, you could simply type the entire API endpoint into the
> browser address bar and receive the XML (or JSON-formatted, if you provide
> a header requesting it) response from the server directly. If you are more
> comfortable on the command-line or in an IDE, you can use Java, Groovy,
> etc. or even curl to invoke the API purely through code. There are also API
> tools (no endorsement, but some are listed here [1]) for hitting these APIs
> from a GUI application.
>
> Here is a good starting point for using the Network tab of the developer
> tools window [2].
>
> Glad to hear you have been able to write some custom processors. Let us
> know if you have any additional questions.
>
> [1]
> https://stackoverflow.com/questions/4486658/what-tools-do-you-use-to-test-your-public-rest-api
> [2] http://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art034
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 7:42 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
>
> (Make that "not much of a web page developer". I'm a back-end Java guy
> mostly, don't much value my UI skills and so have stayed away from the
> browser except for straight HTML and CSS.)
>
> On Fri, Jul 8, 2016 at 8:39 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
>
> Thanks, Andy and Matt, I had actually tried out the Developer Tools
> earlier today as suggested by the doc, but, I'm not much of a web
> developer, don't have a lot of familiarity with those tools in Chrome or in
> Firefox, and I couldn't see anything going on (in Firefox) that was
> helpful. I may have to grow strong that way.
>
> For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
> appearance, we can move up, but you point out something we're already
> beginning to pay attention to: incompatiblities. Still we're committed to
> (and love!) the technology. I've written a dozen custom processors (for
> proprietary ETL stuff) already and my colleague has done the same.
>
> Thanks, again.
>
>
> On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example”  on the canvas,
> I get this response:
>
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> 
> 
> 392b161b-d9cc-442a-9394-8db9cee33ba7
> 
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> 0 / 0 bytes
> NiFi Flow
> 0 / 0 bytes
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> b642a34d-c18c-4bb1-b836-6bbc9d32f31e
> 0 / 0 bytes
> *Token Example*
> 0 / 0 bytes
> 0 bytes
> Invalid
> 0
> 00:00:00.000
> *Base64EncodeContent*
> 0 bytes
> 
> 0 / 0 bytes
> 0
> 0 bytes
> 0 bytes
> 0 / 0 bytes
> 0 / 0 bytes
> 16:09:46 

Re: A few NiFi ReST questions...

[Russell Bateman]

Thanks, Andy and Matt, I had actually tried out the Developer Tools earlier
today as suggested by the doc,but, I'm not much of a web developer, don't
have a lot of familiarity with those tools in Chrome or in Firefox, and I
couldn't see anything going on (in Firefox) that was helpful. I may have to
grow strong that way.

For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
appearance, we can move up, but you point out something we're already
beginning to pay attention to: incompatiblities. Still we're committed to
(and love!) the technology. I've written a dozen custom processors (for
proprietary ETL stuff) already and my colleague has done the same.

Thanks, again.


On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example”  on the canvas,
> I get this response:
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> 
> 
> 392b161b-d9cc-442a-9394-8db9cee33ba7
> 
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> 0 / 0 bytes
> NiFi Flow
> 0 / 0 bytes
> 
> 0
> d17d7a95-6301-46ff-b003-11c58698bd31
> b642a34d-c18c-4bb1-b836-6bbc9d32f31e
> 0 / 0 bytes
> *Token Example*
> 0 / 0 bytes
> 0 bytes
> Invalid
> 0
> 00:00:00.000
> *Base64EncodeContent*
> 0 bytes
> 
> 0 / 0 bytes
> 0
> 0 bytes
> 0 bytes
> 0 / 0 bytes
> 0 / 0 bytes
> 16:09:46 PDT
> 0 / 0 bytes
> 0 bytes
> 
> 
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <matt.c.gil...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bate...@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
>  1. How many times a processor processed anything?
>  2. How many flowfiles were processed?
>  3. How many flowfiles were produced?
> 

Re: A few NiFi ReST questions...

[Russell Bateman]

Ah, for (9), I used system-diagnostics. (I had not spelled it correctly
when trying it before.)

On Fri, Jul 8, 2016 at 3:35 PM, Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
>1. How many times a processor processed anything?
>2. How many flowfiles were processed?
>3. How many flowfiles were produced?
>4. Can I get this information per processor?
>5. How can I tell different instances of the same processor apart?
>6. Why can't I see the name I gave a processor in configuration? (For
>example, I named an instance of GetFile to "Get PDF files from test 
> fodder".
>7. How can I get a list of items on the NiFi canvas like processors
>and relationships.
>8. How many times processors or a processor failed?
>9. What is the profile of resource usage, like memory in use?
>10. What is the profile of processor latency?
>   - flowfiles backed up awaiting processing
>   - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>

A few NiFi ReST questions...

[Russell Bateman]

I'm trying to figure out how to use the ReST (nifi-api) interface to
accomplish a number of things. First, I've played successfully with it
doing simple things like getting configuration, a list of existing
processors, and the like. I've even discovered that in
controller/history/processors/{processorId} I can use the processor name
for {processorId} instead of an intangible or impossible number.

What puzzles me is that I need a {process-group-id} to do so many things.
I'm not using a process group, my flow is very simple for now. Maybe, think
I, I have one process group and I just need its id, but how do I get that?
I see no list API for it.

The long list of things I'm hoping to do down the road from a UI is to be
able to see things like:

   1. How many times a processor processed anything?
   2. How many flowfiles were processed?
   3. How many flowfiles were produced?
   4. Can I get this information per processor?
   5. How can I tell different instances of the same processor apart?
   6. Why can't I see the name I gave a processor in configuration? (For
   example, I named an instance of GetFile to "Get PDF files from test fodder".
   7. How can I get a list of items on the NiFi canvas like processors and
   relationships.
   8. How many times processors or a processor failed?
   9. What is the profile of resource usage, like memory in use?
   10. What is the profile of processor latency?
  - flowfiles backed up awaiting processing
  - flowfiles in other states (?)

Any nudge on any of this would be very welcome. NiFi is new enough that
there's precious few samples of people using the ReST interface.

Thanks

Re: Required, either-or properties

[Russell Bateman]

We'll make it work!

Thanks.

On Thu, May 12, 2016 at 11:56 AM, Joe Witt <joe.w...@gmail.com> wrote:

> Russ,
>
> Yeah - I recommend you mark them both optional then use customValidate
> to check for 'should never happen cases' and then provide validation
> errors for this scenarios which explain to the user proper handling.
> You can also use the PropertyDescriptor of each property to explain
> its intended relationship to other properties.
>
> Does that seem like it will take care of your case?
>
> Thanks
> Joe
>
> On Thu, May 12, 2016 at 1:35 PM, Russell Bateman
> <russell.bate...@perfectsearchcorp.com> wrote:
> > Joe,
> >
> > Thanks for your reply.
> >
> > As I'm thinking about it, validation of the property value isn't so much
> my
> > problem. It's documentation.
> >
> > If I mark the property documentation for both properties as required,
> then
> > my consumer will wonder what supplying both would mean. However, one of
> the
> > two, but never both is required. If both are supplied (whatever that
> would
> > mean in the mind of the consumer), I ignore the template on the
> filesystem
> > path since I check for the existence of the direct content property
> first.
> >
> > Is this dilemma a candidate for the /customValidate/ method you mention?
> >
> > Best,
> > Russ
> >
> >
> > On 05/12/2016 11:28 AM, Joe Witt wrote:
> >>
> >> Russell,
> >>
> >> Validators on property descriptors help with validating that property
> >> alone.  But in the processor API there is 'customValidate' method you
> >> can implement which is used to do things like compound/conditional
> >> validation.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, May 12, 2016 at 1:25 PM, Russell Bateman
> >> <russell.bate...@perfectsearchcorp.com> wrote:
> >>>
> >>> How are folk specifying processor properties in the case where one of
> two
> >>> properties is required, but not both? I'm just wondering if there's a
> >>> best
> >>> practice here or must I say something in the description?
> >>>
> >>> For example, I've written a processor that implements Apache Velocity
> >>> templating. I require the template content be passed either directly as
> >>> the
> >>> value of a property, "Template content", or a filesystem path to this
> >>> content in a property, "Template filepath".
> >>>
> >
>

Re: Required, either-or properties

[Russell Bateman]

Joe,

Thanks for your reply.

As I'm thinking about it, validation of the property value isn't so much 
my problem. It's documentation.


If I mark the property documentation for both properties as required, 
then my consumer will wonder what supplying both would mean. However, 
one of the two, but never both is required. If both are supplied 
(whatever that would mean in the mind of the consumer), I ignore the 
template on the filesystem path since I check for the existence of the 
direct content property first.


Is this dilemma a candidate for the /customValidate/ method you mention?

Best,
Russ

On 05/12/2016 11:28 AM, Joe Witt wrote:

Russell,

Validators on property descriptors help with validating that property
alone.  But in the processor API there is 'customValidate' method you
can implement which is used to do things like compound/conditional
validation.

Thanks
Joe

On Thu, May 12, 2016 at 1:25 PM, Russell Bateman
<russell.bate...@perfectsearchcorp.com> wrote:

How are folk specifying processor properties in the case where one of two
properties is required, but not both? I'm just wondering if there's a best
practice here or must I say something in the description?

For example, I've written a processor that implements Apache Velocity
templating. I require the template content be passed either directly as the
value of a property, "Template content", or a filesystem path to this
content in a property, "Template filepath".

Required, either-or properties

[Russell Bateman]

How are folk specifying processor properties in the case where one of 
two properties is required, but not both? I'm just wondering if there's 
a best practice here or must I say something in the description?


For example, I've written a processor that implements Apache Velocity 
templating. I require the template content be passed either directly as 
the value of a property, "Template content", or a filesystem path to 
this content in a property, "Template filepath".

Re: @DynamicProperty annotation of more than one property?

[Russell Bateman]

I'm volunteering to write if the help is wanted, however, I don't have 
the bandwidth to become much more deeply involved though perhaps, as 
time went on, I would.


However, I can help. Meanwhile, I'm digging writing NiFi processors very 
much.


Russ

On 04/19/2016 09:24 AM, dan bress wrote:

Russell,
I agree the syntax for multiple DynamicProperties is a little awkward,
and that our documentation could be improved in this situation(I didn't see
reference to @DynamicProperty in the developer doc).

Also Java8 supports the concept of repeatable annotations[1], which
would erase the need for the @DynamicProperties by allowing you to specify
multiple @DynamicProperty annotations on a single class.  Since NiFi
appears to be running on Java8, I think we should create a ticket to update
the documentation generation code to handle repeated annotations, and
update our processors/controller services/reporting tasks to use them.  We
should also create a ticket to update the developer documentation to
describe the annotations usage.

[1] http://docs.oracle.com/javase/tutorial/java/annotations/repeating.html

Dan

On Tue, Apr 19, 2016 at 7:23 AM Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:


Thanks, Joe. This worked.

You know, I'm just the sort of guy that would add stuff like that to the
documentation if the way were smoothed to it. The problem as I see it
with NiFi doc is that it's really good the way it is and if we add
more--and we should--it's going to get longer and harder to read.
Someone should slightly redesign it with links to subordinate documents
to avoid making it too complicated yet give folk a place to go for
deeper help. I would be happy to help out once such a thing as that is
decided.

Russ

On 04/15/2016 06:39 PM, Joe Percivall wrote:

Hello Russell,

The annotation you are looking for is @DynamicProperties[1] an example

of it in use is in the PutFTP processor[2].

[1]

https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/DynamicProperties.java

[2]

https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFTP.java#L50


Hope that helps,
Joe

- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Friday, April 15, 2016 7:46 PM, Russell Bateman <

russell.bate...@perfectsearchcorp.com> wrote:



What's the syntax for defining more than one dynamic property for a
processor? I need to specify up to three distinctly different ones and
attempting to do it all in

@DynamicProperty(
   name = "{blah,blah2,blah3}",
   value = "{\"blah-value\",\"blah2-value\",\"blah3-value\"}",
supportsExpressionLanguage = false,
   description = "Big blah, blah, blah..."
)

is under-powered.

Many thanks.

Re: @DynamicProperty annotation of more than one property?

[Russell Bateman]

Thanks, Joe. This worked.

You know, I'm just the sort of guy that would add stuff like that to the 
documentation if the way were smoothed to it. The problem as I see it 
with NiFi doc is that it's really good the way it is and if we add 
more--and we should--it's going to get longer and harder to read. 
Someone should slightly redesign it with links to subordinate documents 
to avoid making it too complicated yet give folk a place to go for 
deeper help. I would be happy to help out once such a thing as that is 
decided.


Russ

On 04/15/2016 06:39 PM, Joe Percivall wrote:

Hello Russell,

The annotation you are looking for is @DynamicProperties[1] an example of it in 
use is in the PutFTP processor[2].

[1] 
https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/DynamicProperties.java
[2] 
https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutFTP.java#L50


Hope that helps,
Joe

- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Friday, April 15, 2016 7:46 PM, Russell Bateman 
<russell.bate...@perfectsearchcorp.com> wrote:



What's the syntax for defining more than one dynamic property for a
processor? I need to specify up to three distinctly different ones and
attempting to do it all in

@DynamicProperty(
  name = "{blah,blah2,blah3}",
  value = "{\"blah-value\",\"blah2-value\",\"blah3-value\"}",
supportsExpressionLanguage = false,
  description = "Big blah, blah, blah..."
)

is under-powered.

Many thanks.

@DynamicProperty annotation of more than one property?

[Russell Bateman]

What's the syntax for defining more than one dynamic property for a 
processor? I need to specify up to three distinctly different ones and 
attempting to do it all in


@DynamicProperty(
name = "{blah,blah2,blah3}",
value = "{\"blah-value\",\"blah2-value\",\"blah3-value\"}",
supportsExpressionLanguage = false,
description = "Big blah, blah, blah..."
)

is under-powered.

Many thanks.

Re: Multiple nar/custom processors: advisable directory structure

[Russell Bateman]

Welcome Idioma...

1) You'll want to subsume your new processors under deeper Java packages 
(you probably knew that).


2) In addition to the Java code, you'll add:
-src
  -main
- resources
  - META-INF
- services
  - org.apache.nifi.processor.Processor
containing a list of the package paths to each of your new processors.

Does this help?

On 04/14/2016 09:20 AM, idioma wrote:

Hi,
currently, I have one custom processor + test in a similar folder structure
in my IDE (IntelliJ):

-CustomProcessors
-nifi-myprocessor-nar
-nifi-myprocessor
   -src
   -main
   -java
   MyProcessor.java
   -test
   -MyProcessorTest.java

I am now in the process to add another processor, what is the best approach?
Shall I have 2 new folders for the nar and one containing the actual
processor? I would like to generate a basic structure for the processor (as
it describes here:
https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html).
Is that advisable when adding another custom processor?

Thanks,




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Multiple-nar-custom-processors-advisable-directory-structure-tp9089.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: catch commit error in OnTrigger to diversify session behaviour

[Russell Bateman]

Oleg,

Agreed. As I started only a few months ago, I have been using 
AtomicReference and it has been reliable and satisfied all my needs. 
(Just sayin'.)


Best

On 04/14/2016 05:22 AM, Oleg Zhurakousky wrote:

A bit unrelated, but how do you guys feel if we deprecate ObjectHolder so it 
could be gone by 1.0?
AtomicReference is available from Java 5

Cheers
Oleg


On Apr 14, 2016, at 5:18 AM, Bryan Bende  wrote:

Hello,

It may be easier to move the load() out of the InputStreamCallback. You
could do something like this...

final ObjectHolder holder = new ObjectHolder(null);

session.read(flowFile, new InputStreamCallback() {

@Override
public void process(InputStream in) throws IOException {
StringWriter strWriter = new StringWriter();
IOUtils.copy(in, strWriter, "UTF-8");
String contents = strWriter.toString();
holder.set(contents);
}
});

try {
load(holder.get());
session.transfer(flowFile, SUCCESS);
  } catch (IOException e) {
session.transfer(flowFile, FAILURE);
}


-Bryan

On Thu, Apr 14, 2016 at 9:06 AM, idioma  wrote:


Hi,
I have modified my onTrigger in this way:

session.read(flowFile, new InputStreamCallback() {

@Override
public void process(InputStream in) throws IOException {

StringWriter strWriter = new StringWriter();
IOUtils.copy(in, strWriter, "UTF-8");
String contents = strWriter.toString();

try {
load(contents);
} catch (IOException e) {
e.getMessage();
boolean error = true;
throw e;
}
}
});

What I am struggling with is how to send it to a failure or a success
depending on the error being thrown. Any help would be appreciated, thank
you so much.



--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/catch-commit-error-in-OnTrigger-to-diversify-session-behaviour-tp9027p9062.html
Sent from the Apache NiFi Developer List mailing list archive at
Nabble.com.

Re: Can't add an attribute inside session read call-back

[Russell Bateman]

Hope not to make copy/paste mistakes. I can share the code snippets, but 
unfortunately not a trace. I didn't keep one and my efforts to restore 
the broken code only leave me frustrated. (I've obliterated my NAR 
several times, but the debugger tells me that the lines don't match up 
when I step through, can't set breakpoints, etc. Sorry.)


How I was doing it:

  public void onTrigger( final ProcessSession session, final 
DataExtractor filter, final String regexPattern )

  throws ProcessException
  {
final FlowFile flowfile = session.get();
*final AtomicReference< FlowFile > flowFileHolder = new 
AtomicReference<>();**

**flowFileHolder.set( flowfile );*

session.read( flowfile, new InputStreamCallback()
{
  @Override
  public void process( InputStream in ) throws IOException
  {
RegularExpressionMatch match;

try
{
  // slurp the in-coming stream to match into...
  Matcher matcher = Pattern.compile( regexPattern ).matcher( 
StreamUtilities.slurp( in, -1 ) );

  if( !matcher.find() )
throw new IOException( "Failed to match regular expression 
pattern in the document" );

  match = new RegularExpressionMatch( matcher );
}
catch( Exception e )
{
  throw new IOException( "Failed to create regular expression 
matcher with passed properties" );

}

Tagtag  = filter.buildTag( match );
String xml;
xml = ( Utilities.isEmpty( tag ) )
  ? "(No tag built for this document.)"
  : new TagUtilities( tag, true /*properties.debug*/ 
).getXmlFromTag( 0 );

*   try**
**   {**
** FlowFile flowFileWithAttributes = flowFileHolder.get();**
** flowFileWithAttributes = session.putAttribute( 
flowFileWithAttributes, "concept", xml );**

** flowFileHolder.set( flowFileWithAttributes );**
   }
   catch( Throwable e )
   {
->   e.printStackTrace();  // IllegalStateException
   }
*  }
} );

// the flowfile now has attributes we put on it...
session.transfer( flowFileHolder.get(), new 
ProcessorRelationships().getSuccess() );

  }

How I'm doing it now:

   public void onTrigger( final ProcessSession session, final
   DataExtractor filter, final String regexPattern )
throws ProcessException
   {
  FlowFile flowfile = session.get();
   *  final Stringattribute;*
   *  final AtomicReference< String > attributeHolder = new
   AtomicReference<>();*

  session.read( flowfile, new InputStreamCallback()
  {
@Override
public void process( InputStream in ) throws IOException
{
  RegularExpressionMatch match;

  try
  {
// slurp the in-coming stream to match into...
String incoming = StreamUtilities.slurp( in, -1 );
Matcher matcher = Pattern.compile( regexPattern ).matcher(
   incoming );
if( !matcher.find() )
  throw new IOException( "Failed to match regular
   expression pattern in the document" );
match = new RegularExpressionMatch( matcher );
  }
  catch( Exception e )
  {
throw new IOException( "Failed to create regular expression
   matcher with passed properties" );
  }

  Tagtag  = filter.buildTag( match );
  String xml;
  xml = ( Utilities.isEmpty( tag ) )
? "(No tag built for this document.)"
: new TagUtilities( tag, true /*properties.debug*/
   ).getXmlFromTag( 0 );
   *  attributeHolder.set( xml );*
}
  } );

  /* Can't change anything about the session right inside here:
   don't change flowfiles
   * in the callback! Set the attribute here instead of inside the
   read!
   */
   *  attribute = attributeHolder.get();*
   *  flowfile  = session.putAttribute( flowfile, "concept", attribute );*

  // the flowfile now has attributes we put on it...
  session.transfer( flowfile, new
   ProcessorRelationships().getSuccess() );
   }

On 03/23/2016 03:42 PM, Oleg Zhurakousky wrote:

Russell

This doesn’t sound right. Would you care to share a code snippet on how you set 
the attribute as well as stack trace?

Cheers
Oleg


On Mar 23, 2016, at 5:33 PM, Russell Bateman 
<russell.bate...@perfectsearchcorp.com> wrote:

I stumbled upon something peculiar in NiFi. I had been attaching an attribute 
to a flowfile in the session.read() call-back. The NiFi unit testing framework 
tolerated this, but when I finally ran my processor for real, it blew chunks 
(IllegalStateException). I solved the problem by saving the new attribute and 
attaching it instead outside the call-back just before calling 
session.transfer().

The only reason I'm pointing this out is b

Re: Processor additional documentation

[Russell Bateman]

Uwe,

When I asked this question a few days ago. What you're doing is rolling 
your whole processor into a NAR instead of JARing it first, then rolling 
the JAR into a NAR. This doesn't work and still won't work when they fix 
it (not just for the reason of /additional//Details.html/ because 
there's other stuff that doesn't work either). So don't do it that way.


   Russell,

   There is a Maven archetype that will generate the project structure
   for you:

   
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-MavenProcessorArchetype

   That page also shows how the bundle should be laid out. Let us know
   if it doesn't make sense.

   -Bryan


On 03/17/2016 02:30 PM, Uwe Geercken wrote:

Hello,

I am writing my first processor.

As described in the documentation, I have added an HTML file to be used when the user 
selects "Usage":

docs/com.datamelt.nifi.test.TemplateProcessor/additionalDetails.html

This is located in the root or the Processors nar file.

The processor class is this:

com/datamelt/nifi/test/TemplateProcessor.class

The processor works, but selecting "Usage" won't show my HTML file.

I understood that I write the HTML file and Nifi will picks it up when it 
starts. Or is this not true?

Thanks for feedback,

Uwe

Re: User friendly versus system friendly Question

[Russell Bateman]

Uwe,

While my usage is surely unrelated to yours, I do exactly this thing 
which is to process a CSV producing as many new output flowfiles as 
there are lines and the names of these flowfiles are, in my case, 
conveniently chosen based on the significant content of each line (so no 
problem coming up with names as there is a person's name and other 
discriminating particulars in the result).


I have another processor that analyzes in-coming XML files and breaks 
them each into several flowfiles according to dates associated in the 
content I'm separating out. Again, no difficulty choosing easily 
recognizable names for the output flowfiles (using hints from the 
original flowfile name plus the resulting content date).


In terms of the pipeline, this all works very well moving on to other 
processors as it does on its way to a database.


Not sure this reply helps, but it describes what I do.

Another very cool thing one gets almost for free with NiFi is an arc 
(relationship) via which the original document can continue along a 
different path, get stored, processed or whatever I want to do with it. 
This is something that was less gracefully provided for in our previous 
pipeline scheme.


Hope this helps.

Cheers


On 03/17/2016 08:09 AM, Uwe Geercken wrote:

Hello,
  
my first mailing here. I am a Java developer, using Apache Velocity, Drill, Tomcat, Ant, Pentaho ETL, MongoDb, Mysql and more and I am very much a data guy.
  
I have used Nifi for a while now and started yesterday of coding my first processor. I basically do it to widen my knowledge and learn something new.
  
I started with the idea of combining Apache Velocity - a template engine - with Nifi. So in comes a CSV file, it gets merged with a template containing formatting information and some placeholders (and some limited logic maybe) and out comes a new set of data, formatted differently. So it separates the processing logic from the formatting. One could create HTML, XML, Json or other text based formats from it. Easy to use and very efficient.
  
Now my question is: Should I rather implement the logic this way that I process a whole CSV file - which usually has multiple lines? That would be good for the user as he or she has to deal with only one processor doing the work. But the logic would be more specialized.
  
The other way around, I could code the processor to handle one row of the CSV file and the user will have to come up with a flow that divides the CSV file into multiple flowfiles before my processor can be used. That is not so specialized but it requires more preparation work from the user.
  
I tend to go the second way. Also because there is already a processor that will split a file into multiple flowfiles. But I wanted to hear your opinion of what is the best way to go. Do you have a recommendation for me? (Maybe the answer is to do both?!)
  
Thanks for sharing your thoughts.
  
Uwe