[
https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022298#comment-13022298
]
Doug Cutting commented on AVRO-806:
-----------------------------------
I was thinking of just creating columns for the fields of the fields of the
top-level record. In this approach, a union would be written as a union,
prefixed with a varint indicating the branch taken.
If we stored union branches separately then we'd also need a column that has
the varint. Iterators would then use this to decide when a column has a value.
For nested unions I think the iterators would need to have a list of pointers
to varints.
The use case is to accelerate scans of a subset of fields. Further
acceleration is possible if things are columnized more deeply, but we probably
want to stop at some fixed depth in each block regardless. So I'm effectively
proposing a depth of 1. Increasing the depth increases the number of buffer
pointers and the complexity of row iteration. I don't have a clear sense of
when that becomes significant. One way to limit the depth would be to specify
a maximum number of columns, and use a breadth-first walk of the schema until
that number of columns are encountered. However I wonder whether we're
over-engineering this.
> add a column-major codec for data files
> ---------------------------------------
>
> Key: AVRO-806
> URL: https://issues.apache.org/jira/browse/AVRO-806
> Project: Avro
> Issue Type: New Feature
> Components: java, spec
> Reporter: Doug Cutting
>
> Define a codec that, when a data file's schema is a record schema, writes
> blocks within the file in column-major order. This would permit better
> compression and also permit efficient skipping of fields that are not of
> interest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
