[
https://issues.apache.org/jira/browse/AVRO-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920544#action_12920544
]
Doug Cutting commented on AVRO-679:
-----------------------------------
> encoding a block of <int,string,long> as a record array<int>, array<long>,
> array<string> might give a 3-6x increase in decode speed
That's what I meant by a schema transformation. You're transforming
[<int,string,long>] to <[int],[string],[long]>. This might be done
automatically by a layer that implements DatumReader and DatumWriter. The
actual schema of the datafile would be <[int],[string],[long]> but application
code would treat it as [<int,string,long>].
> Improved encodings for arrays
> -----------------------------
>
> Key: AVRO-679
> URL: https://issues.apache.org/jira/browse/AVRO-679
> Project: Avro
> Issue Type: New Feature
> Components: spec
> Reporter: Stu Hood
> Priority: Minor
>
> There are better ways to encode arrays of varints [1] which are faster to
> decode, and more space efficient than encoding varints independently.
> Extending the idea to other types of variable length data like 'bytes' and
> 'string', you could encode the entries for an array block as an array of
> lengths, followed by contiguous byte/utf8 data.
> [1] group varint encoding: slides 57-63 of
> http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/people/jeff/WSDM09-keynote.pdf
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
