Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/934#issuecomment-44899107
Hey Andre, thanks for looking into this. I'm concerned about using Byte
Arrays to represent bytes as I'm worried we will break compatibility when we
start supporting repeated types. Also, the allocation of an array is pretty
expensive.
How does Hive or Impala store Bytes / Shorts in parquet? I think we want
to match whatever they are doing. In the worst case we might consider just
promoting them to Ints, since I assume zigzag encoding is used under the covers
anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
