Re: eliminate nifi-parent, split out nifi-nar-maven-plugin, have nifi in its own tree

[Ryan Blue]

+1

I think separate git repos is a great idea. One thing to clarify, too: 
most of the time the nifi project relies on the last 
nifi-nar-maven-plugin release, right? So that should be transparent for 
most people building the project. It would only be awkward for someone 
updating the maven plugin and testing it out locally because the develop 
branch should always track a release.


Speaking of the develop branch... what about using master like most 
projects after this change?


rb

On 08/10/2015 07:32 AM, Joe Witt wrote:

Team,

We've seen and heard the confusion of folks trying to build NiFi's
goofy three step build process with parent, nar plugin, and nifi.  I
propose to do the following:

1) Eliminate the nifi-parent by pushing anything necessary back into
nifi-nar-maven-plugin.  The DRY concept is valid but just not worth a
third project at this point given how little it avoids meaningful
repetition on.

2) Create a new apache git repo for 'nifi-maven-plugins' and move the
'nifi-nar-maven-plugin' content into it.

3) Remove the nifi-parent and nifi-nar-maven-plugin from nifi folder
and promote the current 'nifi' sub folder to the top level.

Why: Folks are confused as to why they need to build all three and it
is odd that in a given project folder you would have to each manually.
It is just not a generally appreciated fact that you cannot have a
dependency on a maven plugin within the same reactor build that uses
that builds that plugin.  By cleaning this up people can just download
the source and build it.  We don't have to have any protracted build
cycles for 'nifi maven plugings' anymore leaving dependency on a
snapshot in the nifi tree.

If there seems to be consensus on this i'll put in the infra ticket soon.

Thanks
Joe




--
Ryan Blue
Software Engineer
Cloudera, Inc.

RE: [DISCUSS] Feature proposal: Read-only mode as default

[Mark Payne]

I'm definitely a +1. I accidentally drag processors all the time when I'm 
panning around a large graph.

I can understand how someone would get annoyed with this, though, and I can 
also appreciate the desire
to not start storing user preferences. However, I think we should probably at 
least supply a system-level
configuration for whether or not to have read-only the default mode. Then 
administrators can turn it on
or off, depending on the users of the system.




 Date: Sat, 8 Aug 2015 20:50:07 -0400
 Subject: Re: [DISCUSS] Feature proposal: Read-only mode as default
 From: a...@adamtaft.com
 To: dev@nifi.apache.org

 Thinking about it some more, I don't mind the concept of read only
 toggle. Maybe it's not set by default, but it could be a really easy UI
 element to add somewhere. Just a little slider-toggle element. [1]

 In theory, this might be a UI only feature, it wouldn't strictly need
 support in the backend api (just guessing). The logic is seemingly already
 there, similar user experience as non-DFMs.

 Anyway, +1 to the concept of read-only toggle mode. -1 for making it
 default, but the user interface element should be easy to work either way.

 Also agree that undo support might be free if versioning is added.

 [1] http://turbo.premiumpixels.com/wp-content/uploads/2011/04/preview2.jpg




 On Sat, Aug 8, 2015 at 3:05 PM, Joe Witt joe.w...@gmail.com wrote:

 Ryan - the other useful case for read-only is basically when is
 scanning around the graph and accidentally moves a processor or
 relationship. By no means a big deal. The idea here was to make it
 explicit though that the user wishes to go into an edit mode.

 I do think the undo mechanism plays well and you're right that we can
 just focus on tightening up the delete case.

 Sounds like the prevailing view is to avoid read-only as a mode but
 rather to make it more explicit whenever we delete - and potentially
 move we could make more specific rather than simply them having
 clicked and dragged which is ambiguous with the process of panning.

 On Sat, Aug 8, 2015 at 2:57 PM, Ryan Blue b...@cloudera.com wrote:
 I'm not a big fan of having a read-only mode by default. It sounds like
 something that would be frustrating for users when they try to make
 changes
 and then have to figure out how to switch modes.

 I think a clearer picture of the problem we're trying to solve would
 help my
 understanding. I'm primarily thinking of the delete case you mentioned
 with
 these comments...

 If we're talking about accidentally deleting processors, then the current
 mechanisms (IIRC) work pretty well: not deleting a running processor, one
 that has live incoming connections, etc. If those rules are
 insufficient, I
 would explore extending them rather than having a global read-only mode.

 For the case where the wrong processor is selected because it is off the
 screen, maybe having the confirmation pop up if anything affected wasn't
 displayed to confirm? That way we don't have confirmations all the time
 but
 still don't do unexpected things.

 I really like the idea of undo as well. If that is limited to
 processors
 that weren't running (because you can't delete ones that are), then that
 makes the undo operation easier to implement.

 rb


 On 08/08/2015 11:31 AM, Joe Witt wrote:

 I can dig the user pref aspect but it would mean we start storing user
 prefs which is a bummer.
 On Aug 8, 2015 1:42 PM, Tony Kurc trk...@gmail.com wrote:

 Personally not a fan of the idea. Maybe something analogous to
 something
 like 'lock the taskbar' in Windows that can have a system default
 setting
 and a user preference of on or off.

 On Sat, Aug 8, 2015 at 11:44 AM, johny casanova 
 computertech2...@gmail.com
 wrote:

 I agree it is easy to move it delete something by mistake. Some flows
 are
 huge or are using,more resources and are slower to load and you can
 accidently do something by mistake. I believe the are yous sure u
 want

 to

 delete? its a good start.
 On Aug 7, 2015 10:31 PM, Joe Witt joe.w...@gmail.com wrote:

 Team,

 We've been hearing from users of nifi that it is too easy to
 accidentally move something on the flow or delete large portions of
 the flow. This is the case when panning the screen for example and
 accidentally having a processor selected instead.

 What is worth consideration then is the notion of making the flow
 'read-only' by default. In this case the user would need to
 explicitly 'enable edit mode'. We would then also support a
 confirmation dialog or similar construct whenever deleting components
 on the flow.

 Anyone have a strong objection to this concept? If so, do you have
 an
 alternative in mind that would help avoid accidental movement?

 Thanks
 Joe






 --
 Ryan Blue
 Software Engineer
 Cloudera, Inc.

  

Re: Help required with my custom controller service

[Brandon DeVries]

All,

Was this ever solved / explained?  I'm having a similar issue.  If there's
a simple answer that I'm missing that would be great... otherwise I might
post an example set of NARs somewhere tomorrow.  Probably the simplest
question first is, if I want to create a custom processor that uses a
SSLContextService, what dependencies do I need to add where (assuming we're
using the processor NAR archetype)?  I've created an example that compiles,
loads in NiFi, and can be dropped on the graph.  However, it can not be
configured to use any currently existing SSLContextService, and any that
are created have the same UUID issue described by David.  Thoughts?

Brandon



On Tue, Jul 28, 2015 at 10:50 AM Mark Payne marka...@hotmail.com wrote:

 Dave,

 In addition to the points that Aldrin brought up, are your CacheService
 interface
 and StandardCacheService classes in the same NAR?

 If they are not in the same NAR, does your NAR for StandardCacheService
 have a Maven dependency
 on the NAR that houses CacheService?

 Thanks
 -Mark

 
  From: aldrinp...@gmail.com
  Date: Mon, 27 Jul 2015 23:26:58 -0400
  Subject: Re: Help required with my custom controller service
  To: dev@nifi.apache.org; davidrsm...@btinternet.com
 
  Dave,
 
  Some quick notes as I work through your setup.
 
  Did you create an entry in your
  NAR's org.apache.nifi.controller.ControllerService file? This would be in
  src/main/resources/META-INF/services directory of your project. It seems
  like this isn't likely the case, but a point that can sometimes get
  overlooked.
 
  Your accessing of the service property in your processor looks good.
 
  Did you accidentally override AbstractControllerService#getIdentifier()?
  It seems like this could lead to the error you are seeing, but it is hard
  to be sure.
 
  If you are able to share more code with us, we can provide some more
  pointed assistance. Have you managed to get your processor and controller
  service running within the test framework [1]? If not, I would suggest
  doing so to cut down on iterations in debugging your problems as well as
  providing a means for you to easily step through the code.
 
  [1]
 http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing
 
  On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH davidrsm...@btinternet.com
 
  wrote:
 
  Hi
  I have been trying to write a simple Cache Controller Service that uses
 a
  Map to hold String, String key/value pairs. I am using
  nifi-0.1.0-incubating. I have created an Interface called CacheService
  which extends ControllerService.public interface CacheService extends
  ControllerService {
 
  I have created an implementation called StandardCacheService which is as
  follows:
  public class StandardCacheService extends AbstractControllerService
  implements CacheService {
  I have compiled the StandardCacheService and created a nar which I have
  then put into my NiFi lib directory, my StandardCacheService Controller
  Service appears in the Controller services tab of the NiFi Flow Settings
  panel, I can edit it and enable it. Also there are no abnormal log
 messages.
  However, when I try to configure the CacheService in my test Processor,
  again I get no errors but when I cannot see my StandardCacheService. My
  processor has the following snippets to implement the Cache
  Service/Controller:
  public static final PropertyDescriptor CACHE_SERVICE = new
  PropertyDescriptor.Builder()
  .name(Cache Service)
  .description(The Controller Service to use in order to obtain a
  Cache Service)
  .required(false)
  .identifiesControllerService(CacheService.class)
  .build();
 
  final CacheService cache =
 
 context.getProperty(CACHE_SERVICE).asControllerService(CacheService.class);
 
 
 
 
  When configuring the processor, the 'Cache Service' drop down doesn't
 show
  a populated list showing my StandardCacheService, but it does allow me
 to
  create a new service, the resulting panel allows me to then select
  StandardCacheService. When I select 'create' I get a UUID value in the
  Value property of the Configure Processor panel. When I then apply this
  the processor shows a yellow triangle, when I hover over it I get the
  following message:
  'Cache Service validated against UUID is invalid because Invalid
  Controller service UUID is not a valid Controller service Identifier
 or
  does not reference the correct type of Controller service.'
  Can you help please
  Many thanksDave
 
 

Re: Instantiating a Controller Service in a Junit test

[DAVID SMITH]

Mark
Thanks for the information, it works a treat.
Dave
 


 On Monday, 10 August 2015, 1:12, Mark Payne marka...@hotmail.com wrote:
   

 David,

Yes, you'll also need to set the controller service in your processor. Sorry, I 
forgot to mention that.

So after the call to runner.enableControllerService(), and before the call to 
runner.run(), you would do:

runner.setProperty(CacheTester.CACHE_SERVICE, my-cache);

This way, the CacheTester processor knows to reference that controller service. 
So your method will look like:

@Test
public void checkCache() throws InitializationException, IOException{
   final TestRunner runner = TestRunners.newTestRunner(CacheTester.class);
   final StandardCacheService cacheService = new StandardCacheService();
   runner.addControllerService(my-cache, cacheService);
   runner.setProperty(cacheService, StandardCacheService.DATAFILE, 
/data/TEST_FILE);
   runner.setProperty(CacheTester.CACHE_SERVICE, my-cache);
   runner.enableControllerService(cacheService);
   runner.run();
}

Thanks
-Mark


 Date: Sun, 9 Aug 2015 21:06:18 +
 From: davidrsm...@btinternet.com
 To: dev@nifi.apache.org
 Subject: Re: Instantiating a Controller Service in a Junit test

 Mark
 Thanks for the reply, I have changed my test as you suggested, see below:
 @Test
 public void checkCache() throws InitializationException, IOException{
 final TestRunner runner = TestRunners.newTestRunner(CacheTester.class);
 final StandardCacheService cacheService = new StandardCacheService();
 runner.addControllerService(my-cache, cacheService);
 runner.setProperty(cacheService, StandardCacheService.DATAFILE, 
 /data/TEST_FILE);
 runner.enableControllerService(cacheService);
 runner.run();
 }


 When I run my test I now get a null pointer exception in my CacheTester 
 class. It appears the cache in my CacheTester class doesn't exist, when I 
 comment out all the calls to the cache methods the test passes.
 If I understand the code above correctly I don't believe I have set the 
 PropertyDescriptor in my CacheTester processor class which is shown below has 
 been set, am I correct?:
 public static final PropertyDescriptor CACHE_SERVICE = new 
 PropertyDescriptor.Builder() .name(Cache Service) .description(The 
 Controller Service to use in order to obtain a Cache Service) 
 .required(false) .identifiesControllerService(CacheServiceAPI.class) .build();


 BTW, the former I mentioned in my original post was referring to the 
 descriptions I had given about how to instantiate the Controller Service.
 Many thanksDave


 On Sunday, 9 August 2015, 21:05, Mark Payne marka...@hotmail.com wrote:


 Hi David,

 You should be able to just import your StandardCacheService in your unit test.

 You can then instantiate the controller service and use 
 TestRunner.addControllerService, as you're doing here.
 At that point, to set the properties, you can use TestRunner.setProperty. For 
 example:

 final StandardCacheService cacheService = new StandardCacheService();
 runner.addControllerService(my-cache, cacheService);
 runner.setProperty(cacheService, StandardCacheService.DATAFILE, /data/file);
 runner.enableControllerService(cacheService);

 There is no need to actually create the Logger and call initialize, as that 
 is handled for you when you call TestRunner.addControllerService.

 In your message, can you explain a bit further what you meant by
 If the former is correct how do I set the PropertyDescriptor as when I did 
 try this option the StandardCacheService.DATAFILE PropertyDescriptor was 
 never visible?

 It's important that you not mark the PropertyDescriptor as private, or else 
 you won't be able to access it, and you'll also want to ensure that
 it is returned by your getSupportedPropertyDescriptors() method. If I am 
 misunderstanding the comment, please advise.

 Let me know if this clears things up for you, or if you need any more details.

 If anything doesn't make sense, just give a shout - we're always happy to 
 help! :)

 Thanks
 -Mark


 
 Date: Sun, 9 Aug 2015 14:40:53 +
 From: davidrsm...@btinternet.com
 To: dev@nifi.apache.org
 Subject: Instantiating a Controller Service in a Junit test

 Hi
 I have written a simple Cache Controller Service, this Controller Service 
 has a property which if populated allows the cache to be populated when it 
 is intialized. I have also written a simple processor that allows me to 
 utilize the Controller Service and checks some of the preloaded values and 
 also checks some of the cache methods.
 I now want to write some Junit tests for my processor, and I want to 
 instantiate my Cache Controller Service. I have looked at other Junit test 
 classes in the nifi-0.2.1 source release for some guidance on how to do 
 this, looking particularly at the test classes for the DetectDuplicate 
 processor.
 I have imported the Controller Service API and based on what I saw in the 
 

RE: Help required with my custom controller service

[Mark Payne]

Brandon,

I've seen this occur when I forgot to add a dependency on the 
ssl-context-service-api-nar in my NAR's pom.

You should have something like:
dependency
  groupIdorg.apache.nifi/groupId
  artifactIdssl-context-service-api/artifactId
  version0.3.0-SNAPSHOT/version
  typenar/type
/dependency

Is that by chance the issue, or is there something else going on?

Thanks
-Mark


 From: b...@jhu.edu
 Date: Mon, 10 Aug 2015 19:13:50 +
 Subject: Re: Help required with my custom controller service
 To: dev@nifi.apache.org

 All,

 Was this ever solved / explained? I'm having a similar issue. If there's
 a simple answer that I'm missing that would be great... otherwise I might
 post an example set of NARs somewhere tomorrow. Probably the simplest
 question first is, if I want to create a custom processor that uses a
 SSLContextService, what dependencies do I need to add where (assuming we're
 using the processor NAR archetype)? I've created an example that compiles,
 loads in NiFi, and can be dropped on the graph. However, it can not be
 configured to use any currently existing SSLContextService, and any that
 are created have the same UUID issue described by David. Thoughts?

 Brandon



 On Tue, Jul 28, 2015 at 10:50 AM Mark Payne marka...@hotmail.com wrote:

 Dave,

 In addition to the points that Aldrin brought up, are your CacheService
 interface
 and StandardCacheService classes in the same NAR?

 If they are not in the same NAR, does your NAR for StandardCacheService
 have a Maven dependency
 on the NAR that houses CacheService?

 Thanks
 -Mark

 
 From: aldrinp...@gmail.com
 Date: Mon, 27 Jul 2015 23:26:58 -0400
 Subject: Re: Help required with my custom controller service
 To: dev@nifi.apache.org; davidrsm...@btinternet.com

 Dave,

 Some quick notes as I work through your setup.

 Did you create an entry in your
 NAR's org.apache.nifi.controller.ControllerService file? This would be in
 src/main/resources/META-INF/services directory of your project. It seems
 like this isn't likely the case, but a point that can sometimes get
 overlooked.

 Your accessing of the service property in your processor looks good.

 Did you accidentally override AbstractControllerService#getIdentifier()?
 It seems like this could lead to the error you are seeing, but it is hard
 to be sure.

 If you are able to share more code with us, we can provide some more
 pointed assistance. Have you managed to get your processor and controller
 service running within the test framework [1]? If not, I would suggest
 doing so to cut down on iterations in debugging your problems as well as
 providing a means for you to easily step through the code.

 [1]
 http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing

 On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH davidrsm...@btinternet.com

 wrote:

 Hi
 I have been trying to write a simple Cache Controller Service that uses
 a
 Map to hold String, String key/value pairs. I am using
 nifi-0.1.0-incubating. I have created an Interface called CacheService
 which extends ControllerService.public interface CacheService extends
 ControllerService {

 I have created an implementation called StandardCacheService which is as
 follows:
 public class StandardCacheService extends AbstractControllerService
 implements CacheService {
 I have compiled the StandardCacheService and created a nar which I have
 then put into my NiFi lib directory, my StandardCacheService Controller
 Service appears in the Controller services tab of the NiFi Flow Settings
 panel, I can edit it and enable it. Also there are no abnormal log
 messages.
 However, when I try to configure the CacheService in my test Processor,
 again I get no errors but when I cannot see my StandardCacheService. My
 processor has the following snippets to implement the Cache
 Service/Controller:
 public static final PropertyDescriptor CACHE_SERVICE = new
 PropertyDescriptor.Builder()
 .name(Cache Service)
 .description(The Controller Service to use in order to obtain a
 Cache Service)
 .required(false)
 .identifiesControllerService(CacheService.class)
 .build();

 final CacheService cache =

 context.getProperty(CACHE_SERVICE).asControllerService(CacheService.class);




 When configuring the processor, the 'Cache Service' drop down doesn't
 show
 a populated list showing my StandardCacheService, but it does allow me
 to
 create a new service, the resulting panel allows me to then select
 StandardCacheService. When I select 'create' I get a UUID value in the
 Value property of the Configure Processor panel. When I then apply this
 the processor shows a yellow triangle, when I hover over it I get the
 following message:
 'Cache Service validated against UUID is invalid because Invalid
 Controller service UUID is not a valid Controller service Identifier
 or
 does not reference the correct type of Controller service.'
 Can you help please
 Many