[
https://issues.apache.org/jira/browse/AVRO-3451?focusedWorklogId=744758&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744758
]
ASF GitHub Bot logged work on AVRO-3451:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Mar/22 19:29
Start Date: 20/Mar/22 19:29
Worklog Time Spent: 10m
Work Description: martin-g commented on pull request #1608:
URL: https://github.com/apache/avro/pull/1608#issuecomment-1073316938
Here are the results from the benchmark on my laptop:
`master` branch:
```
10000,1,Small,0.039890636,0.042860779
10000,1,Big,0.099197248,0.103504636
1,100000,Small,4.622638324,20.889562055
100,1000,Small,0.52317468,0.731093513
10000,10,Small,0.530728866,0.633507069
1,100000,Big,10.907742395,71.910063022
100,1000,Big,1.151314511,2.06744156
10000,10,Big,1.172123896,1.171496815
```
This PR:
```
10000,1,Small,0.011186142,0.043883047
10000,1,Big,0.056111559,0.114635545
1,100000,Small,4.730861091,21.375156375
100,1000,Small,0.170062576,0.701255961
10000,10,Small,0.116927097,0.453378823
1,100000,Big,10.054628933,69.582575465
100,1000,Big,0.651133748,2.037338469
10000,10,Big,0.702202783,1.152962118
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 744758)
Remaining Estimate: 20m (was: 0.5h)
Time Spent: 40m (was: 0.5h)
> fix poor Avro write performance
> -------------------------------
>
> Key: AVRO-3451
> URL: https://issues.apache.org/jira/browse/AVRO-3451
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
> Affects Versions: 1.11.0
> Environment: Mac OS X Big Sur
> {code:java}
> installed toolchains
> --------------------
> stable-x86_64-apple-darwin (default)
> nightly-x86_64-apple-darwin
> active toolchain
> ----------------
> stable-x86_64-apple-darwin (default)
> rustc 1.56.1 (59eed8a2a 2021-11-01) {code}
> Reporter: Kevin
> Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2022-03-14 at 7.30.24 PM.png
>
> Original Estimate: 1h
> Time Spent: 40m
> Remaining Estimate: 20m
>
> Rust implementation of Apache Avro library – apache-avro (née avro-rs) –
> demonstrates poor write performance when serializing Rust structures to Avro.
> Profiling indicates that this implementation spends an inordinate amount of
> time in the function {{encode::encode_ref}} performing {{clone()}} and
> {{drop}} operations related to a HashMap<String, Schema> type.
> We modified the function {{encode_ref0}} as follows:
> {code:java}
> -pub fn encode_ref(value: &Value, schema: &Schema, buffer: &mut Vec<u8>) {
> - fn encode_ref0(
> +pub fn encode_ref<'a>(value: &Value, schema: &'a Schema, buffer: &mut
> Vec<u8>) {
> + fn encode_ref0<'a>(
> value: &Value,
> - schema: &Schema,
> + schema: &'a Schema,
> buffer: &mut Vec<u8>,
> - schemas_by_name: &mut HashMap<String, Schema>,
> + schemas_by_name: &mut HashMap<&'a str, &'a Schema>,
> ) {
> match &schema {
> Schema::Ref { ref name } => {
> - let resolved =
> schemas_by_name.get(name.name.as_str()).unwrap();
> + let resolved = schemas_by_name.get(&name.name as
> &str).unwrap();
> return encode_ref0(value, resolved, buffer, &mut
> schemas_by_name.clone());
> }
> Schema::Record { ref name, .. }
> | Schema::Enum { ref name, .. }
> | Schema::Fixed { ref name, .. } => {
> - schemas_by_name.insert(name.name.clone(), schema.clone());
> + schemas_by_name.insert(&name.name, &schema);
> }
> _ => (),
> }{code}
> to remove any need for Clone in the {{schemas_by_name}} cache and see a
> notable improvement (factor of 4 to 5) in our application with this change.
> After this change, all Cargo Tests still pass and Benchmarks display a very
> significant improvement in Write performance across the board. Attached below
> is one example benchmark for {{big schema, write 10k records}} with Before on
> the Left and After on the Right.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
