Hi Chris and Mark
Many thanks for your reply. You are totally right (of course :-) ) and that
is also the knowledge and understanding I had (have). except that the EL
only will be evaluated using variable registry. Sorry for that :-)
My goal is to only have one flow file, with all records from
Thanks Chris. That’s exactly right.
Given that you’re seeing the Max Bin Age is the cause, the solution would be to
increase the max bin age if you want fewer FlowFiles.
The data is merged when any one of the following conditions is met:
- Minimum Number of Records is reached AND Minimum Bin
For “Minimum Number of Records”, the docs [1] indicate that the field does
support Expression Language but "will be evaluated using variable registry
only”, i.e. it doesn’t use FlowFile attributes, which it appears you’re trying
to do in your example within this email chain.
If you provenance
Hi
By degreasing the batch size for the SiteToSiteStatusReportingTask I get
even more flowfiles. So just for testing I now have total of 9 files
(2.75MB) in the incomming queue to the mergeRecord.
The total number of records above 2000, so I have set the "Minimum Number
of Records" to 1500 and the
Hey Mark
I tried another idea to dynamically set the "Minimum Number of Records" by
EL. Editing the field it says that EL is supported, so I tried this:
${record.count:minus(1):multiply(3)}
But the processor does not like this:
Perform Validation
nifi.mydomain.com:8443 - Component is invalid:
Hey Mark
Many thanks for your reply. But it's in fact the Details field which does
not help me.
At 08:16:00 all 3 nodes generate a SiteToSiteStatusReport.
At 08:16:11.003 the MergeRecords have a JOIN event. Joining 2 files:
"Records Merged due to: Bin has reached Max Bin Age"
At 08:16:11.008 the
Hey Jens,
My recommendation is to take a look at the data provenance for MergeRecord
(i.e., right-click on the Processor and go to Data Provenance.) Click the
little ‘i’ icon on the left for one of the JOIN events.
There, it will show a “Details” field, which will tell you why it merged the
Hi all
I'm running a 3 node cluster at version 1.16.2. I'm using
the SiteToSiteStatusReportingTask to monitor and check for any
backpressures or queues. I'm trying to merge all 3 reports into 1, but must
of the times I always get 2 flowfile after my MergeRecord.
To be sure the nodes are creating
