Joe Witt created NIFI-16379:
-------------------------------
Summary: Restore GenerateFlowFile non-unique generation and batch
write behavior
Key: NIFI-16379
URL: https://issues.apache.org/jira/browse/NIFI-16379
Project: Apache NiFi
Issue Type: Test
Affects Versions: 2.12.0
Reporter: Joe Witt
Assignee: Joe Witt
NIFI-16322 changed GenerateFlowFile to stream generated content instead of
allocating a full-size byte array. The change introduced substantial performance
and behavioral regressions for the processor's primary load-testing use case.
Before NIFI-16322, when Unique FlowFiles was false, GenerateFlowFile generated
content once during scheduling and reused that content for every invocation.
Each FlowFile in Batch Size was independently written to the Content Repository.
After NIFI-16322, the processor stores only a random seed. Every invocation
regenerates the complete payload through an 8 KB buffer, even though non-unique
FlowFiles contain identical content. Text generation performs a Random.nextInt()
operation for every generated byte. This substantially reduces throughput for
the default non-unique configuration.
NIFI-16322 also changed Batch Size behavior. The processor now writes the first
FlowFile and creates the remaining FlowFiles using ProcessSession.clone(). As a
result, a batch performs one Content Repository write instead of one write per
FlowFile and produces clone behavior rather than independent generated content.
This makes GenerateFlowFile unsuitable for measuring Content Repository write
throughput and changes observable provenance behavior. DuplicateFlowFile already
exists for workloads that require cloning.
Restore the pre-NIFI-16322 behavior:
- Generate and cache content once when Unique FlowFiles is false.
- Independently write every FlowFile requested by Batch Size.
- Remove the ProcessSession.clone() batch optimization.
- Restore the documented higher-throughput behavior for non-unique content.
- Retain the bounded File Size validator introduced by NIFI-16322 so values
greater than Integer.MAX_VALUE remain invalid and cannot be truncated.
Tests should verify:
- Non-unique content remains identical across triggers.
- Unique content differs between generated FlowFiles.
- Batch Size produces independently written FlowFiles rather than shared-content
clones.
- File Size greater than Integer.MAX_VALUE remains invalid.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
