sankalpsthakur opened a new pull request, #1649:
URL: https://github.com/apache/arrow-go/pull/1649
### Rationale for this change
A Parquet file that uses a logical type added after this version of arrow-go
makes `file.OpenParquetFile` panic with `invalid logical type`. The thrift
decoder skips the union member it does not know and leaves an empty
`LogicalType` union behind. `getLogicalType`
(`parquet/schema/logical_types.go`) has a case for every member, and nothing
for the empty one, so it reaches `panic("invalid logical type")`. The old `case
l == nil` could never run, because `l.IsSetSTRING()` dereferences the nil union
first.
The format says what a reader should do (parquet-format `LogicalTypes.md`,
"Unsupported Logical Types"): "readers should ignore both the logical type
annotation and column order for that column. Only the physical type information
should be used to process the column's data." Arrow C++ does this since
apache/arrow#41765. arrow-go is one of the implementations listed without a fix
in apache/parquet-format#599.
### What changes are included in this PR?
- `getLogicalType` returns `NoLogicalType` for a nil union and for an empty
one. The dead `case l == nil` branch is removed.
- Logical type: a primitive column with an unknown annotation reads as
having no logical type, so it is processed by its physical type.
- Column order: `PrimitiveNodeFromThrift` marks such a node, and
`Column.SortOrder()` returns `SortUNKNOWN` for it. Without this the column
became `NoLogicalType`, which maps to the default order of the physical type
(unsigned for BYTE_ARRAY), and the file's min/max statistics were trusted even
though the unknown type may define another order. Arrow C++ makes the sort
order UNKNOWN as well.
- A group node with an unknown annotation is built with no logical type.
- `converted_type`: behaviour is unchanged. If the element also has a
`converted_type` next to the unknown logical type (for example UTF8 or LIST),
it is dropped with the annotation, and the node reads as plain physical type or
plain group. A newer logical type has no converted equivalent, so the case is
rare. I kept the PR to the logical type path, like #41765, and can fall back to
the converted type in a follow-up if you prefer that.
- Out of scope: `getLogicalType` still panics for a TIME or TIMESTAMP whose
`TimeUnit` union is empty, and an unrecognized logical/physical type
combination still returns an error from `NewPrimitiveNodeLogical`. Writing such
a file back drops the unknown annotation, because `NoLogicalType` is not
serialized.
### Are these changes tested?
Yes.
- `TestUnknownThriftLogicalType`
(`parquet/schema/unknown_logical_type_test.go`) covers a primitive and a group
element with and without a `converted_type`, and asserts `SortUNKNOWN` for the
primitive column. It fails without the sort order change (expected 2, actual 1).
- `TestReadUnknownLogicalType`
(`parquet/pqarrow/unknown_logical_type_test.go`) reads
`unknown-logical-type.parquet` from parquet-testing (a string column and a
binary column) with a checked allocator.
- `TestInvalidConvertedTypeInDeserialize` used to expect the panic and now
expects the node to read as having no logical type.
Run:
- `PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema
./parquet/pqarrow -run 'Unknown|Invalid' -count=1`: ok
- `PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema
-count=1`: ok
- `go vet ./parquet/schema ./parquet/pqarrow` and `gofmt -l`: clean
### Are there any user-facing changes?
Yes. Files that use a logical type from a newer writer can be opened and
read by their physical type instead of panicking. Min/max statistics of such a
column are not used for pruning. There is no API change.
### AI/LLM disclosure
- AI coding tools were used to help draft or modify code and this PR
description.
- I reviewed the complete change, understand the reasoning, and ran the
reported local tests before submitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]