[
https://issues.apache.org/jira/browse/AVRO-3451?focusedWorklogId=744766&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744766
]
ASF GitHub Bot logged work on AVRO-3451:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Mar/22 20:08
Start Date: 20/Mar/22 20:08
Worklog Time Spent: 10m
Work Description: martin-g merged pull request #1608:
URL: https://github.com/apache/avro/pull/1608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 744766)
Time Spent: 1h 20m (was: 1h 10m)
> fix poor Avro write performance
> -------------------------------
>
> Key: AVRO-3451
> URL: https://issues.apache.org/jira/browse/AVRO-3451
> Project: Apache Avro
> Issue Type: Improvement
> Components: rust
> Affects Versions: 1.11.0
> Environment: Mac OS X Big Sur
> {code:java}
> installed toolchains
> --------------------
> stable-x86_64-apple-darwin (default)
> nightly-x86_64-apple-darwin
> active toolchain
> ----------------
> stable-x86_64-apple-darwin (default)
> rustc 1.56.1 (59eed8a2a 2021-11-01) {code}
> Reporter: Kevin
> Priority: Major
> Labels: pull-request-available
> Attachments: Screen Shot 2022-03-14 at 7.30.24 PM.png
>
> Original Estimate: 1h
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Rust implementation of Apache Avro library – apache-avro (née avro-rs) –
> demonstrates poor write performance when serializing Rust structures to Avro.
> Profiling indicates that this implementation spends an inordinate amount of
> time in the function {{encode::encode_ref}} performing {{clone()}} and
> {{drop}} operations related to a HashMap<String, Schema> type.
> We modified the function {{encode_ref0}} as follows:
> {code:java}
> -pub fn encode_ref(value: &Value, schema: &Schema, buffer: &mut Vec<u8>) {
> - fn encode_ref0(
> +pub fn encode_ref<'a>(value: &Value, schema: &'a Schema, buffer: &mut
> Vec<u8>) {
> + fn encode_ref0<'a>(
> value: &Value,
> - schema: &Schema,
> + schema: &'a Schema,
> buffer: &mut Vec<u8>,
> - schemas_by_name: &mut HashMap<String, Schema>,
> + schemas_by_name: &mut HashMap<&'a str, &'a Schema>,
> ) {
> match &schema {
> Schema::Ref { ref name } => {
> - let resolved =
> schemas_by_name.get(name.name.as_str()).unwrap();
> + let resolved = schemas_by_name.get(&name.name as
> &str).unwrap();
> return encode_ref0(value, resolved, buffer, &mut
> schemas_by_name.clone());
> }
> Schema::Record { ref name, .. }
> | Schema::Enum { ref name, .. }
> | Schema::Fixed { ref name, .. } => {
> - schemas_by_name.insert(name.name.clone(), schema.clone());
> + schemas_by_name.insert(&name.name, &schema);
> }
> _ => (),
> }{code}
> to remove any need for Clone in the {{schemas_by_name}} cache and see a
> notable improvement (factor of 4 to 5) in our application with this change.
> After this change, all Cargo Tests still pass and Benchmarks display a very
> significant improvement in Write performance across the board. Attached below
> is one example benchmark for {{big schema, write 10k records}} with Before on
> the Left and After on the Right.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
