Phillip LeBlanc created ARROW-17133:
---------------------------------------
Summary: pqarrow: PlainFixedLenByteArrayEncoder behaves
differently from DictFixedLenByteArrayEncoder with null values where schema has
Nullable: false
Key: ARROW-17133
URL: https://issues.apache.org/jira/browse/ARROW-17133
Project: Apache Arrow
Issue Type: Bug
Components: Go, Parquet
Affects Versions: 8.0.0
Reporter: Phillip LeBlanc
I have created a small repro to illustrate this bug:
https://gist.github.com/phillipleblanc/5e3e2d0e6914d276cf9fd79e019581de
When writing a Decimal128 array to a Parquet file the pqarrow package will
prefer to use DictFixedLenByteArrayEncoder. If the size of the array goes over
some threshold, it will switch to using PlainFixedLenByteArrayEncoder.
The DictFixedLenByteArrayEncoder tolerates null values in a Decimal128 array
with the arrow schema set to Nullable: false, however the
PlainFixedLenByteArrayEncoder will not tolerate null values and will panic.
Having null values in an array marked as non-nullable is an issue in the user
code - however, it was surprising that my buggy code was working some times and
not working other times. I would expect the PlainFixedLen encoder to handle
nulls the same way as the DictFixedLen encoder or for the DictFixedLen encoder
to panic.
An observation is that most other array types handle nulls with the schema
marked as non-nullable when writing to Parquet; this was the first instance I
found in the pqarrow package where having the Arrow schema marked as Nullable
was necessary for Parquet writing arrays with null values. Again, debatable if
this is desirable or not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
