Hi Uwe, I believe a lot of this is covered in the Developer Guide [1]. Specifically, there are discussions of various processor patterns, including Split Content [2], and a section on Cohesion and Usability [3], which states:
> In order to avoid these issues, and make Processors more reusable, a > Processor should always stick to the principal of "do one thing and do it > well." Such a Processor should be broken into two separate Processors: one to > convert the data from Format X to Format Y, and another Processor to send > data to the remote resource. I call this the “Unix model” — it is better to join several small, specific tools together to accomplish a task than re-invent larger tools every time a small modification is required. In general, that would lead me to develop processors that operate on the smallest unit of data necessary — a single line, element, or record makes sense — unless more context is needed for completeness, or the performance is so grossly different that it is inefficient to operate on such small quantities. Finally, with regards to your question about which provenance events to use in various scenarios, I agree the documentation is lacking. Luckily Drew Lim did some great work improving this documentation. While it has not been released in an official version, both the Developer Guide [4] and User Guide [5] have received substantial enhancements, describing the complete list of provenance event types and their usage/meaning. This work is available on master and will be released in 1.2.0. The project has certainly evolved over a long lifetime, and you are correct that different processors have different philosophies. Sometimes that is the result of different authors, sometimes it is a legitimate result of the wide variety of scenarios that these processors interact with. Improving the user experience and documentation is always important, and getting started with and maximizing the usefulness of these processors is one of our top priorities. I would also reference Chesterton’s Fence [6] here — there are definitely improvements to be made, I do not disagree. I would also caution against, as I have done in the past, making changes to improve a system without understanding the way it got to the current state. Once one has a firm grasp on the history, then a reasonable plan may be made to improve things. We are always welcoming of suggestions to improve the experience for the community. Hope this helps and I’d love to get your feedback on where else we can be better. Thanks. [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html> [2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#split-content-one-to-many <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#split-content-one-to-many> [3] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#cohesion-and-reusability <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#cohesion-and-reusability> [4] https://github.com/andrewmlim/nifi/blob/bd9eb0ac6009845de9d5a34bd5384ade1945befd/nifi-docs/src/main/asciidoc/developer-guide.adoc#provenance-events <https://github.com/andrewmlim/nifi/blob/bd9eb0ac6009845de9d5a34bd5384ade1945befd/nifi-docs/src/main/asciidoc/developer-guide.adoc#provenance-events> [5] https://github.com/andrewmlim/nifi/blob/bd9eb0ac6009845de9d5a34bd5384ade1945befd/nifi-docs/src/main/asciidoc/user-guide.adoc#data-provenance <https://github.com/andrewmlim/nifi/blob/bd9eb0ac6009845de9d5a34bd5384ade1945befd/nifi-docs/src/main/asciidoc/user-guide.adoc#data-provenance> [6] https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence <https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence> Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 16, 2017, at 3:28 PM, Uwe Geercken <[email protected]> wrote: > > Hello, > > I have a little bit of a hard time to design processors correctly. I find it > difficult to decide if a processor should e.g. process a single line from a > flow file or process also flow files with multiples lines of data (e.g. in > the case of CSV files). Another point is the handling of header rows. One > other point is data provenenance events: what is the correct event I should > use when modifying attributes, content or both? > > Is there a guide which outlines the best practices for such cases? I have the > feeling that many of the processors handle these issues quite differently. I > think there either should be a sort of standard or otherwise it should be > well documented. And although there is very good documentation available for > the project, for some of the processors one has to play around quite a bit to > get it right because they behave differently or have a different philosophie > and one has to understand it first to get it right. > > Would appreciate to get some feedback and advice or pointers to documentation. > > Uwe
signature.asc
Description: Message signed with OpenPGP using GPGMail
