Mark Payne created NIFI-8609:
--------------------------------
Summary: Improve efficiency of converting Record object to Avro
GenericRecord object
Key: NIFI-8609
URL: https://issues.apache.org/jira/browse/NIFI-8609
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Reporter: Mark Payne
Assignee: Mark Payne
During some performance tests and profiling, I found that for a given flow,
pushing Avro records to Kafka, one of the most expensive parts of the flow was
converting our Record (MapRecord) object into a GenericRecord object for the
Avro Writer.
I created a simple unit test to determine a baseline for performance numbers
before making any changes. The unit test creates a Record with 100 null String
fields, half of which have a {{null}} value assigned to them. I then converted
the record into an Avro GenericRecord via
{{AvroTypeUtil.createAvroRecord(record, avroSchema);}} in a loop of 1,000,000
iterations and output how long it took; this was then repeated 1,000 times in
order to allow the JVM to warm up.
Numbers on my Macbook Pro showed after the first few iterations that the amount
of time needed to convert 1 million records was on the order of 4.5 seconds.
After updating the code, performance numbers are just under 2 seconds. So
somewhere on the order of 2x better performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
