[
https://issues.apache.org/jira/browse/AVRO-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AVRO-3156:
---------------------------------
Labels: pull-request-available (was: )
> Performance degradation in SpecificRecordBuilder introduced in 1.9.0
> --------------------------------------------------------------------
>
> Key: AVRO-3156
> URL: https://issues.apache.org/jira/browse/AVRO-3156
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.9.0, 1.10.0
> Environment: Using SpecificData in environments with multiple
> classloaders.
> Reporter: Steven Aerts
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-06-11-14-27-16-689.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The change introduced in spark 1.9.0 which changed:
> {code:java}
> SpecificData.get(){code}
> into:
> {code:java}
> SpecificData.getForSchema(schema){code}
> introduced a significant performance degradation in environments where the
> class of {{schema}} is provided by a different classloader then the
> classloader containing {{SpecificData}}.
>
> A possible solution is to use the {{classCache}} of the default
> {{SpecificData}} so the sometimes expensive classloader codepath is cached.
> (PR coming up)
> We noticed this in after trying out a spark upstep from spark 3.1.0 (avro
> 1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of
> times resolving the same class.
> With this patch this resolving time was brought back from 74% to 0.70%.
> JMC flamegraph showing this issue:
> !image-2021-06-11-14-27-16-689.png!
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
