Sam
#1 will definitely help. Reduces max memory usage of that processor
to at worst:
size of each flowfile (not its content just the metadata) * 5001 *
number of threads
#2 If the content repository is filling up then it is undersized for
how much data can be queued and is not likely going to be better
without archive. Now fixing #1 will help transaction rate and
throughput so it might be better anyway but ensure your content repo
can hold at least as much data as you want to be able to buffer at any
one time.
#3/#4 No definitely not necessary. I'd remove this line from the
custom processor as it takes the full content and turns it into an
attribute
String results = value.get();
if (results != null && !results.isEmpty()) {
flowFile = session.putAttribute(flowFile, "json", results);
}
Thanks
Joe
On Mon, Feb 13, 2017 at 11:47 AM, sam <[email protected]> wrote:
> Thanks for that information! Still trying to understand Nifi better.
>
> 1. SplitText (5000 'Line Split Count') -> SplitText (1 'Line Split Count') I
> added that, will see if that resolves the problem.
>
> 2. Usually breaks after running couple of hours, when content repository is
> full. I have disabled archiving, which should be fine in my case since
> reading data from persistent storage.
>
> 3. Events are very small, hardly 500 bytes. So actually after SplitText
> there are around hundred thousands of tiny flowfiles.
>
> 4. Is it necessary to promote the entirety of the content into a flow file
> attribute?
> I am directly not even using flowfile attributes, only using the content
> (these readymade processors I list are the only ones using it). Can still be
> a concern? or can I remove some part of flowfiles?
>
> 5. I know this can happen again, I want to be able to track which s3 files
> failed in case something breaks, so that I can replay them. Any integration
> for provenance repo to something like cloudwatch would have been great to
> monitor failed files. Is there any similar processor?
>
> Appreciate your help.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Nifi-in-a-hung-state-tp14713p14724.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
