Hi Phil, We are also doing a similar thing but not keeping all the content after each transformation externally. What we do is, only send the flow file attributes to an external storage (like file / Event Hub / Database/NoSQL) using AttributesToJSON processor and then send it for logging after every logical step where we want to log (after adding couple of additional details like - step name, #of rows in file, hascode etc.).
For your scenario, I think you can simply clone the output relationship from each of your processors and send it to a single/multiple logging/sink processors. For keeping the lineage, you have couple of options: 1. Use different sink/folder/table for each step (with corresponding name) 2. Keep file name consistent to track the lineage 3. Modify the Flow file content to make sure you can track the lineage from the metadata content. Regards, Manish -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Tuesday, September 27, 2016 7:33 PM To: [email protected] Subject: logging all transformed flowfiles Hello, My SW context : standalone NiFi 1.0.0 My Problem : I would like to log all the different transformations applied to an initial file ( input) up to exiting the DF ( output) : If imagine this simple DF : File1 (in) --> Processor1 --> flow1 --> Processor2 --> flow2 --> File2 (out) I would like to store outside of Nifi ( in my own external DB) -> File1, flow1, flow2, File2 Are there some simple REST API to help to accomplish this ( I looked at Data provenance and SiteToSiteProvenanceReportingTask but not clearly found the right way to implement this) Any idea ? Phil Best regards -----Original Message----- From: Joe Witt [mailto:[email protected]] Sent: mercredi 30 mars 2016 04:58 To: [email protected] Subject: Re: Developing dataflows in the canvas editor Dmitry these are great questions and Chris that was in my opinion a pretty excellent response - 'noob' or not. The only thing I'd add Dmitry is that some of what you're saying regarding templates themselves is very true. We can do better and so much more than we are. We have a feature proposal/discussion framing here [1,2] and please by all means help us shape how this evolves. [1] https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows [2] https://cwiki.apache.org/confluence/display/NIFI/Extension+Registry Thanks Joe On Tue, Mar 29, 2016 at 1:59 PM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) <[email protected]> wrote: > Dimitri, > > From one noob to another, welcome. > > All modifications to the canvas are automatically saved. If you want to > organize multiple flow instances look to process groups. Drag a process > group onto the canvas. Double click the process group to open it. Then drag a > template onto the canvas. Use the breadcrumbs to navigate back to the root > process group (root of the canvas). Create a second process group. Wash and > repeat. Process groups can be nested to your hearts content. Process groups > themselves can be saved as templates. You can also copy then paste in > process groups. And you can drag processors and process groups into other > process groups, although I am not sure that you can do this with > multi-select. They are great for creating a high-level abstraction for a > complex flow. > > I find its best to use the zoom controls. For what its worth Google Maps > uses the same paradigm for zooming. I’m not sure these web-apps can really > understand “gestures”, its just that the browser translates the gesture into > scroll events which NiFi uses for zooming. > > Good luck, > > Chris > > Date: Tuesday, March 29, 2016 at 1:27 PM > To: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > Subject: Developing dataflows in the canvas editor > From: Dmitry Goldenberg > <[email protected]<mailto:[email protected]>> > Reply-To: "[email protected]<mailto:[email protected]>" > <[email protected]<mailto:[email protected]>> > > Hi, > > These may be just 'noob' impressions from someone who hasn't learned enough > NiFi yet (I may be missing something obvious). > z > My first confusion is about dataflows vs. templates. I've developed a couple > of templates. Now I want to drop a template into the canvas and treat that > as a dataflow or an instance of a template. But I don't see a way to save > this instance into any persistent store, or any way to manage its lifecycle > (edit, delete etc). Is there something I'm missing or are there features in > progress related to this? > > I mean, where does the dataflow go if I kill the browser? It seems to > persist... but what happens when I want to create a slightly different > rendition of the same flow? Is there a namespaced persistence for dataflows > with CRUD operations supported? I keep looking for a File -> New, File -> > Open, File -> Save type of metaphor. > > My second item is the mouse roller. Yes, the mouse roller which at least on > the Mac causes the canvas to zoom in or zoom out. Having used other > products, the typical metaphor seems to be that rolling gets you a vertical > scrollbar and you can scroll up and down, with a separate UI gesture that > lets you zoom in/out. I can't seem to get used to this zooming behavior. > > Thoughts? > > Thanks, > - Dmitry _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
