sankalpsthakur opened a new pull request, #1649:
URL: https://github.com/apache/arrow-go/pull/1649

   ### Rationale for this change
   
   A Parquet file that uses a logical type added after this version of arrow-go 
makes `file.OpenParquetFile` panic with `invalid logical type`. The thrift 
decoder skips the union member it does not know and leaves an empty 
`LogicalType` union behind. `getLogicalType` 
(`parquet/schema/logical_types.go`) has a case for every member, and nothing 
for the empty one, so it reaches `panic("invalid logical type")`. The old `case 
l == nil` could never run, because `l.IsSetSTRING()` dereferences the nil union 
first.
   
   The format says what a reader should do (parquet-format `LogicalTypes.md`, 
"Unsupported Logical Types"): "readers should ignore both the logical type 
annotation and column order for that column. Only the physical type information 
should be used to process the column's data." Arrow C++ does this since 
apache/arrow#41765. arrow-go is one of the implementations listed without a fix 
in apache/parquet-format#599.
   
   ### What changes are included in this PR?
   
   - `getLogicalType` returns `NoLogicalType` for a nil union and for an empty 
one. The dead `case l == nil` branch is removed.
   - Logical type: a primitive column with an unknown annotation reads as 
having no logical type, so it is processed by its physical type.
   - Column order: `PrimitiveNodeFromThrift` marks such a node, and 
`Column.SortOrder()` returns `SortUNKNOWN` for it. Without this the column 
became `NoLogicalType`, which maps to the default order of the physical type 
(unsigned for BYTE_ARRAY), and the file's min/max statistics were trusted even 
though the unknown type may define another order. Arrow C++ makes the sort 
order UNKNOWN as well.
   - A group node with an unknown annotation is built with no logical type.
   - `converted_type`: behaviour is unchanged. If the element also has a 
`converted_type` next to the unknown logical type (for example UTF8 or LIST), 
it is dropped with the annotation, and the node reads as plain physical type or 
plain group. A newer logical type has no converted equivalent, so the case is 
rare. I kept the PR to the logical type path, like #41765, and can fall back to 
the converted type in a follow-up if you prefer that.
   - Out of scope: `getLogicalType` still panics for a TIME or TIMESTAMP whose 
`TimeUnit` union is empty, and an unrecognized logical/physical type 
combination still returns an error from `NewPrimitiveNodeLogical`. Writing such 
a file back drops the unknown annotation, because `NoLogicalType` is not 
serialized.
   
   ### Are these changes tested?
   
   Yes.
   
   - `TestUnknownThriftLogicalType` 
(`parquet/schema/unknown_logical_type_test.go`) covers a primitive and a group 
element with and without a `converted_type`, and asserts `SortUNKNOWN` for the 
primitive column. It fails without the sort order change (expected 2, actual 1).
   - `TestReadUnknownLogicalType` 
(`parquet/pqarrow/unknown_logical_type_test.go`) reads 
`unknown-logical-type.parquet` from parquet-testing (a string column and a 
binary column) with a checked allocator.
   - `TestInvalidConvertedTypeInDeserialize` used to expect the panic and now 
expects the node to read as having no logical type.
   
   Run:
   
   - `PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema 
./parquet/pqarrow -run 'Unknown|Invalid' -count=1`: ok
   - `PARQUET_TEST_DATA=$PWD/parquet-testing/data go test ./parquet/schema 
-count=1`: ok
   - `go vet ./parquet/schema ./parquet/pqarrow` and `gofmt -l`: clean
   
   ### Are there any user-facing changes?
   
   Yes. Files that use a logical type from a newer writer can be opened and 
read by their physical type instead of panicking. Min/max statistics of such a 
column are not used for pruning. There is no API change.
   
   ### AI/LLM disclosure
   
   - AI coding tools were used to help draft or modify code and this PR 
description.
   - I reviewed the complete change, understand the reasoning, and ran the 
reported local tests before submitting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to