[
https://issues.apache.org/jira/browse/AVRO-3156?focusedWorklogId=736570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736570
]
ASF GitHub Bot logged work on AVRO-3156:
----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Mar/22 10:25
Start Date: 04/Mar/22 10:25
Worklog Time Spent: 10m
Work Description: steven-aerts closed pull request #1253:
URL: https://github.com/apache/avro/pull/1253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 736570)
Remaining Estimate: 0h
Time Spent: 10m
> Performance degradation in SpecificRecordBuilder introduced in 1.9.0
> --------------------------------------------------------------------
>
> Key: AVRO-3156
> URL: https://issues.apache.org/jira/browse/AVRO-3156
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.9.0, 1.10.0
> Environment: Using SpecificData in environments with multiple
> classloaders.
> Reporter: Steven Aerts
> Priority: Major
> Attachments: image-2021-06-11-14-27-16-689.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The change introduced in spark 1.9.0 which changed:
> {code:java}
> SpecificData.get(){code}
> into:
> {code:java}
> SpecificData.getForSchema(schema){code}
> introduced a significant performance degradation in environments where the
> class of {{schema}} is provided by a different classloader then the
> classloader containing {{SpecificData}}.
>
> A possible solution is to use the {{classCache}} of the default
> {{SpecificData}} so the sometimes expensive classloader codepath is cached.
> (PR coming up)
> We noticed this in after trying out a spark upstep from spark 3.1.0 (avro
> 1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of
> times resolving the same class.
> With this patch this resolving time was brought back from 74% to 0.70%.
> JMC flamegraph showing this issue:
> !image-2021-06-11-14-27-16-689.png!
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
