Arek Burdach created NIFI-5757:
----------------------------------
Summary: AvroRecordSetWriter synchronize every access to
compiledAvroSchemaCache
Key: NIFI-5757
URL: https://issues.apache.org/jira/browse/NIFI-5757
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Affects Versions: 1.7.1
Reporter: Arek Burdach
Avro record serialization is a quite expensive operation.
This stack trace I very often see in thread dumps:
{noformat}
Thread 48583: (state = BLOCKED)
- org.apache.nifi.avro.AvroRecordSetWriter.compileAvroSchema(java.lang.String)
@bci=9, line=124 (Compiled frame)
-
org.apache.nifi.avro.AvroRecordSetWriter.createWriter(org.apache.nifi.logging.ComponentLog,
org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
@bci=96, line=92 (Compiled frame)
- sun.reflect.GeneratedMethodAccessor183.invoke(java.lang.Object,
java.lang.Object[]) @bci=56 (Compiled frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=43 (Compiled frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
@bci=56, line=498 (Compiled frame)
-
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(java.lang.Object,
java.lang.reflect.Method, java.lang.Object[]) @bci=309, line=89 (Compiled
frame)
- com.sun.proxy.$Proxy100.createWriter(org.apache.nifi.logging.ComponentLog,
org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream)
@bci=24 (Compiled frame)
-
org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(org.apache.nifi.flowfile.FlowFile,
org.apache.nifi.serialization.record.RecordSet,
org.apache.nifi.serialization.RecordSetWriterFactory,
org.apache.nifi.serialization.record.RecordSchema, java.lang.String,
java.lang.String) @bci=71, line=169 (Compiled frame)
-
org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_1_0$1.process(java.io.InputStream)
@bci=94, line=412 (Compiled frame)
{noformat}
The reason why it happens is because {{AvroRecordSetWriter}} synchronizing
every access to cache of compiled schemas.
I've prepared PR that is fixing this issue by using {{ConcurrentHashMap}}
instead: https://github.com/apache/nifi/pull/3111
It is not a perfect fix because it removes cache size limitation which BTW was
hardcoded to {{20}}. Services can be reusable by many flows so such a hard
limit is not a good choice.
What do you think about such an improvement?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
