Lars, The information you're providing from the logs is a pretty important bit of debug data.
This concept of 'CONTENTMISSING' being recorded into the Flow File Repository is NiFI's way of saying "Hey I knew about this flow file but when I tried to access the content it was no longer in the content repository". What I'm suggesting is something outside of NiFi itself removed the content. By default, even when you remove content using the NiFi API it isn't actually deleting the content until it has to and it is asynchronous. Even if you had restarted NiFi during this I don't see how this could occur. Even if you have some bugs in the custom processor implementations the issue you're showing here should not be possible. The only explanation that makes sense to me so far is that the content was actually deleted from within the content repository by something other than NiFi. Can you reproduce the issue? Thanks Joe On Tue, Feb 16, 2016 at 4:58 PM, Lars Francke <[email protected]> wrote: > Any ideas on how to debug this further? > > I know very little about the internals of NiFi but there are obviously > still references to that content and it shouldn't have been deleted. Can > you think of a way I could have done this by accident? > > On Tue, Feb 16, 2016 at 10:35 PM, Joe Witt <[email protected]> wrote: > >> Interesting. What that suggests is the content has been removed from >> the content repo itself. >> >> Thanks >> Joe >> >> On Tue, Feb 16, 2016 at 4:15 PM, Lars Francke <[email protected]> >> wrote: >> > I attached a debugger and checked a few of those FlowFiles that failed >> and >> > searched the logs for those. This is what I found: >> > >> > 2016-02-16 18:28:35,953 INFO [main] >> o.a.n.c.repository.FileSystemRepository >> > Found unknown file >> > /Users/lars/Downloads/nifi-0.5.0/content_repository/103/14556368398 >> > 47-103 (1058303 bytes) in File System Repository; archiving file >> > >> > 2016-02-16 18:42:54,840 WARN [Timer-Driven Process Thread-9] >> > o.a.n.c.r.WriteAheadFlowFileRepository Repository Record >> > >> StandardRepositoryRecord[UpdateType=CONTENTMISSING,Record=StandardFlowFileRecord[uuid=af69ca83-fc03-41f0-91e1-e3d65da54840,claim=StandardContentClaim >> > [resourceClaim=StandardResourceClaim[id=1455636632024-102, >> > container=default, section=102], offset=661978, >> > length=10],offset=0,name=69321836993544,size=10]] is marked to be >> aborted; >> > it will be persisted in the FlowFileRepository as a DELETE record >> > >> > Now I can't remember having done this but it's entirely possible that I >> > restarted NiFi prior to my experiment described above. >> > >> > >> > On Tue, Feb 16, 2016 at 9:16 PM, Joe Witt <[email protected]> wrote: >> > >> >> Lars, >> >> >> >> Definitely look forward to understanding the mechanics here a bit >> >> better of what you're seeing and if you can provide something >> >> reproducible. Even if you have a custom processor the API/Process >> >> Session construct should protect from many of the things that can go >> >> wrong there. Now the content repo will likely be large empty as the >> >> data represents on 888KB of data and it is probably in a relative >> >> small number of files on disk. >> >> >> >> Thanks >> >> joe >> >> >> >> On Tue, Feb 16, 2016 at 2:57 PM, Lars Francke <[email protected]> >> >> wrote: >> >> > Hi Matt, >> >> > >> >> > thanks for the quick response. It's late here so I'll try reproducing >> >> > tomorrow. >> >> > >> >> > Source and destination processors are custom processors. >> >> > This is Nifi 0.5.0 RC3 >> >> > >> >> > NiFi thinks all FlowFiles are still there: <http://imgur.com/isDlRk4> >> >> > >> >> > I'm looking at logs now no ERRORs or WARN that seem suspicious so far >> >> > >> >> > On Tue, Feb 16, 2016 at 8:46 PM, Matthew Clarke < >> >> [email protected]> >> >> > wrote: >> >> > >> >> >> Lars, >> >> >> What version of NiFi are you running? >> >> >> What type of processor was your source processor? >> >> >> What type of processor was the destination processor? >> >> >> I tried reproducing using a GenerateFlowFile to produce ~100k >> >> >> Flowfiles on a connection to an UpdateAttribute processor. I then >> >> stopped >> >> >> the GenerateFlowFile processor , added a funnel, and moved the >> >> connection. >> >> >> I also added another processor feeding that same funnel and routed >> the >> >> >> connection from the funnel back to the UpdateAttribute processor. >> The >> >> >> files moved as expected through the funnnel. >> >> >> >> >> >> Can you reproduce? Any other errors in your app log from >> prior >> >> to >> >> >> completing the connection? >> >> >> >> >> >> Thanks, >> >> >> Matt >> >> >> >> >> >> On Tue, Feb 16, 2016 at 1:15 PM, Lars Francke < >> [email protected]> >> >> >> wrote: >> >> >> >> >> >> > Hi, >> >> >> > >> >> >> > I'm trying to understand what happened and how I can prevent this >> in >> >> the >> >> >> > future. >> >> >> > >> >> >> > The outcome seems to be that all my FlowFiles which were sitting >> in a >> >> >> > connection have been deleted from disk. >> >> >> > >> >> >> > I had a flow with two processors connected via a single connection. >> >> >> > >> >> >> > What I did: >> >> >> > * Stop all Processors >> >> >> > * Add a Funnel >> >> >> > * Add a Processor >> >> >> > * Move destination end of existing connection to funnel (with the >> >> >> existing >> >> >> > FlowFiles) >> >> >> > * Connect new Processor to Funnel >> >> >> > * Connect Funnel to old destination Processor >> >> >> > >> >> >> > The connection between the Funnel and the Destination processor >> still >> >> >> shows >> >> >> > all 90k FlowFiles but the Processor fails on session.read with a >> >> >> > MissingFlowFileException. >> >> >> > >> >> >> > Sure enough my content_repository is mostly empty too. >> >> >> > >> >> >> > Now this isn't so bad because it's only a dev environment but I'd >> >> like to >> >> >> > understand how this could happen. Did I do something wrong? >> >> >> > >> >> >> > Any hints on what to search for in the logs or which place in the >> >> source >> >> >> > code to look? >> >> >> > >> >> >> > Cheers, >> >> >> > Lars >> >> >> > >> >> >> >> >> >>
