Emilio Setiadarma created NIFI-12700:
----------------------------------------
Summary: PutKudu memory optimization for unbatched flush mode
(AUTO_FLUSH_SYNC)
Key: NIFI-12700
URL: https://issues.apache.org/jira/browse/NIFI-12700
Project: Apache NiFi
Issue Type: Improvement
Reporter: Emilio Setiadarma
Assignee: Emilio Setiadarma
The PutKudu processor's existing implementation uses a Map of KuduOperation ->
FlowFile to keep track of which FlowFile was processing when the KuduOperation
was created. This is mapping is eventually used to associate FlowFiles with the
RowError (if any occurs), a mapping that is necessary for transferring
FlowFiles to success/failure relationships or logging failures among other
things.
For very large inputs, Kudu Operation objects can grow very large. There is no
memory leak, but still could cause OutOfMemory issues in very large input data.
There is a possibility to not require the use of a KuduOperation -> FlowFile
map for unbatched flush modes (e.g. when using the AUTO_FLUSH_SYNC flush mode,
where the KuduSession.apply() would have already flushed the buffer before
returning,
[https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html)|https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html]
This Jira attempts to capture the efforts for refactoring PutKudu processor to
make it more memory optimized.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
