[
https://issues.apache.org/jira/browse/NIFI-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248438#comment-16248438
]
Yuri commented on NIFI-4569:
----------------------------
Hello Eric,
concerning your points under the *Development workflow* section. The [NiFi
Registry|https://nifi.apache.org/registry.html] subproject aims to deal with
the workflow management. You can check [the current
efforts|https://issues.apache.org/jira/projects/NIFIREG].
> Brief summary of a newbie journey into NiFi
> -------------------------------------------
>
> Key: NIFI-4569
> URL: https://issues.apache.org/jira/browse/NIFI-4569
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Docker
> Affects Versions: 1.4.0
> Reporter: Eric Chaves
> Priority: Minor
>
> Hi folks,
> As requested at Nifi's User mailing list, I'm compiling a small briefing of
> my experience as a new user working with NiFi. This report includes some
> feedback while I was learning NiFi following some tutorial and articles found
> on the internet.
> In particular, after scouting the internet and reading NiFi's docs, I first
> tried to write some custom flows using scripted processors and scripted
> services.
> Most of the troubles I've faced I attribute to my lack of expertise with
> Java/Groovy plus my small knowledge with NiFi's architecture. I did faced
> some inconsistencies however that may be eased for newcomers like me in
> future releases.
> *Development workflow*
> To write my flows I'm using the official docker image (apache/nifi:1.4.0)
> together with other images like mysql, mongodb and localstack (to simulate
> AWS services).
> It took me a while to properly setup a working folder using docker. When I
> simple run the container without any volume binded to my local host, NiFi
> started properly because all the required files are already present in conf
> dir. This setup works fine for trying NiFi out but is not good as development
> starting point because changes may be lost due to container rebuilt.
> To address this I first attempted to bind volumes for ./conf and ./logs
> folder but this didn't worked as expected because this produced and empty
> ./conf folder that prevents nifi from starting. To get this right was a
> little hard and I only did it after looking at some github's projects to
> compare they're ./conf dir with NiFi's default configuration inside the
> docker image.
> _- *Suggestion #1*: NiFi shoud created default configuration files when no
> file is present in ./conf_
> After getting my docker right I started creating some flows following some
> Hortonworks tutorials and others website (like Matt's blog) and the first
> thing that confused me was the lack of a simple to get a new blank canvas. In
> order to do so I need to stop all components, delete processors and groups,
> ports and etc.
> _- *Suggestion #2*: Add an "new canvas" button to the operations pallete (or
> somewhere else)._
> At this point I started writing some flows that I would like to keep under
> version control and there is no simple way for that. NiFi's canvas is kept in
> a gzip file which is not ideal for version control and the current way of
> exporting a canvas requires us to save it as template and then export the
> template. This process seems ok when we have finished a flow and want to
> export it for future use as an actual template but is not good for small
> commits while working in the flow (WIP commits). A better approach would be
> to have the flowfile as xml directly under source control.
> _- *Suggestion #3*: Enable a configuration mode to kept the current canva as
> pure xml instead of GZipped file._
> *Writing Custom Processors*
> Not much to be said here. The experience was awesome. The developers guide is
> very good and Matt's 3 post 'nifi scripting cookbooks' are priceless. Those
> cookbooks should be added to the default documentation.
> Two things hit me here: I assumed that all JAR used by NiFi's default
> processors were available for use by a script processor (requiring it only to
> be imported) but that is not the case and I needed to re-add some JAR's (like
> javax.mail or AWS java sdk) to the script's modules folder. That was not
> intuitive for me but maybe because I'm not a Java developer. I also had a
> trouble with java loading proper handlers for mime types (something related
> to javax.mail) that blocked me to use Groovy for a custom script write
> multipart mime messages (I ended writing it in python).
> - *Suggestion #4*: Make clear which JAR files are available by default inside
> a script processor and which are not and how to properly configure the system
> class loader.
> *Writing Custom Services*
> I'm working on a custom flow that needs to enrich some data records. Since I
> had successfully wrote some processors scripts I tried to script a custom
> LookupService and after googling around I used two sources as reference: 1)
> An Andy Lopresto script found at gist and 2) the test_lookup_inline.groovy at
> NiFi's source code.
> I made some mistakes and my script lookup was not working so I decided to log
> some info in order to troubleshoot my code but no information was logged.
> That's when I noticed that my docker image was not producing any logs. I read
> the administrator guide to see If I was missing something at bootstrap.conf
> file but my file was ok according to the docs.
> It was only when I scouted the logback.xml file that I noticed an variable
> "bootstrap.conf.dir" somewhere and that hinted me to try adding a "log.dir="
> key to the bootstrap.conf file. At this time I also found that I had the
> wrong permissions on my local log folder which was probably the reason why no
> logs were being written. Since I made both changes at once I can't say for
> sure which one fixed the logs.
> Only once I got the nifi-user.log properly working that I could see the
> exception was being raised (because there wasn't a log object) and that was
> the whole error. Once fixed the scripted lookup worked like a charm.
> Matt explained in the user's list the reasons why there is no log by default
> on controller services and it makes sense however as a user I got a little
> lost because when configuring processors I can set the log level very easily
> but when comes to ControllerServices the dialog has no mention about how or
> where log is done.
> The same thing happens when I was editing my script code and re-running. With
> processors I point the script file and in order to have it reloaded in case
> of changes I only need to stop/start the processor.
> For scripted services I assumed that disabling/enabling them would had the
> same effect but that was not the case. Once enabled I could only had it
> reloaded by stoping my docker container and starting it again. It took me a
> while before I could figured out that my new code was not being loaded and
> instead the previous versions still in use (even after disabling/enabling it).
> - *Suggestion #5*: Improve UX consistency between processor configuration and
> controller services configuration. Allow service's code reload with
> enable/disable and add some link for how to properly log service messages.
> Expose a log object on scripted components by default.
> - *Suggestion #6*: Add a section into developer guide explaining the default
> interfaces (like LookupServices) and how they should be implemented/extended
> Anyway I'm really enjoying NiFi and day after day it's becoming easier to
> understand it's component's model.
> Great work guys and thanks for such a great software!!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
