I attached a debugger and checked a few of those FlowFiles that failed and searched the logs for those. This is what I found:
2016-02-16 18:28:35,953 INFO [main] o.a.n.c.repository.FileSystemRepository Found unknown file /Users/lars/Downloads/nifi-0.5.0/content_repository/103/14556368398 47-103 (1058303 bytes) in File System Repository; archiving file 2016-02-16 18:42:54,840 WARN [Timer-Driven Process Thread-9] o.a.n.c.r.WriteAheadFlowFileRepository Repository Record StandardRepositoryRecord[UpdateType=CONTENTMISSING,Record=StandardFlowFileRecord[uuid=af69ca83-fc03-41f0-91e1-e3d65da54840,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1455636632024-102, container=default, section=102], offset=661978, length=10],offset=0,name=69321836993544,size=10]] is marked to be aborted; it will be persisted in the FlowFileRepository as a DELETE record Now I can't remember having done this but it's entirely possible that I restarted NiFi prior to my experiment described above. On Tue, Feb 16, 2016 at 9:16 PM, Joe Witt <[email protected]> wrote: > Lars, > > Definitely look forward to understanding the mechanics here a bit > better of what you're seeing and if you can provide something > reproducible. Even if you have a custom processor the API/Process > Session construct should protect from many of the things that can go > wrong there. Now the content repo will likely be large empty as the > data represents on 888KB of data and it is probably in a relative > small number of files on disk. > > Thanks > joe > > On Tue, Feb 16, 2016 at 2:57 PM, Lars Francke <[email protected]> > wrote: > > Hi Matt, > > > > thanks for the quick response. It's late here so I'll try reproducing > > tomorrow. > > > > Source and destination processors are custom processors. > > This is Nifi 0.5.0 RC3 > > > > NiFi thinks all FlowFiles are still there: <http://imgur.com/isDlRk4> > > > > I'm looking at logs now no ERRORs or WARN that seem suspicious so far > > > > On Tue, Feb 16, 2016 at 8:46 PM, Matthew Clarke < > [email protected]> > > wrote: > > > >> Lars, > >> What version of NiFi are you running? > >> What type of processor was your source processor? > >> What type of processor was the destination processor? > >> I tried reproducing using a GenerateFlowFile to produce ~100k > >> Flowfiles on a connection to an UpdateAttribute processor. I then > stopped > >> the GenerateFlowFile processor , added a funnel, and moved the > connection. > >> I also added another processor feeding that same funnel and routed the > >> connection from the funnel back to the UpdateAttribute processor. The > >> files moved as expected through the funnnel. > >> > >> Can you reproduce? Any other errors in your app log from prior > to > >> completing the connection? > >> > >> Thanks, > >> Matt > >> > >> On Tue, Feb 16, 2016 at 1:15 PM, Lars Francke <[email protected]> > >> wrote: > >> > >> > Hi, > >> > > >> > I'm trying to understand what happened and how I can prevent this in > the > >> > future. > >> > > >> > The outcome seems to be that all my FlowFiles which were sitting in a > >> > connection have been deleted from disk. > >> > > >> > I had a flow with two processors connected via a single connection. > >> > > >> > What I did: > >> > * Stop all Processors > >> > * Add a Funnel > >> > * Add a Processor > >> > * Move destination end of existing connection to funnel (with the > >> existing > >> > FlowFiles) > >> > * Connect new Processor to Funnel > >> > * Connect Funnel to old destination Processor > >> > > >> > The connection between the Funnel and the Destination processor still > >> shows > >> > all 90k FlowFiles but the Processor fails on session.read with a > >> > MissingFlowFileException. > >> > > >> > Sure enough my content_repository is mostly empty too. > >> > > >> > Now this isn't so bad because it's only a dev environment but I'd > like to > >> > understand how this could happen. Did I do something wrong? > >> > > >> > Any hints on what to search for in the logs or which place in the > source > >> > code to look? > >> > > >> > Cheers, > >> > Lars > >> > > >> >
