1fanwang opened a new pull request, #3803:
URL: https://github.com/apache/parquet-java/pull/3803
### Rationale for this change
A 16-byte fixed-size Arrow binary field comes back as variable-width Binary
after writing a file with the converted Parquet schema. The bytes and nulls
survive, but the declared width is lost.
### What changes are included in this PR?
Positive fixed-size binary widths now map to fixed-length Parquet storage
and back. Zero-width fields keep their BINARY representation because Parquet
requires a positive fixed length. Decimal annotations and INT96 mappings are
unchanged.
### Are these changes tested?
The regression writes a real file containing a 16-byte value and a null row,
reads both back, and checks the recovered Arrow schema.
#### Testing Done
| Property | Before | After |
| --- | --- | --- |
| Parquet physical type | BINARY | FIXED_LEN_BYTE_ARRAY |
| Declared byte width | Absent | 16 |
| Recovered Arrow type | Binary | FixedSizeBinary |
I ran the same regression on the baseline and this change with Java 17 and
Maven 3.9.16. From the PR checkout:
```bash
git worktree add -b 1fannnw/repro-fixed-binary \
../parquet-fixed-binary-before 2df8d02678dab4bb8b926a0d3221cc652984c7ab
git diff 2df8d02678dab4bb8b926a0d3221cc652984c7ab HEAD \
-- parquet-arrow/pom.xml \
parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java
|
git -C ../parquet-fixed-binary-before apply -
mvn -B -T2 -f ../parquet-fixed-binary-before/pom.xml \
-pl parquet-arrow -am test \
'-Dtest=TestSchemaConverter#testFixedBinaryFileRoundTrip' \
-Dsurefire.failIfNoSpecifiedTests=false
mvn -B -T2 -pl parquet-arrow -am test \
'-Dtest=TestSchemaConverter#testFixedBinaryFileRoundTrip' \
-Dsurefire.failIfNoSpecifiedTests=false
```
<details>
<summary>Raw logs</summary>
Before:
```text
stored_type=BINARY
stored_width=0
restored_type=Binary
rows=2, payload_preserved=true, null_preserved=true
expected: Schema<fingerprint: FixedSizeBinary(16)>
but was: Schema<fingerprint: Binary>
```
After:
```text
stored_type=FIXED_LEN_BYTE_ARRAY
stored_width=16
restored_type=FixedSizeBinary
rows=2, payload_preserved=true, null_preserved=true
```
</details>
The added dependencies are test-scoped for the file roundtrip.
- [x] Local code review completed
### Are there any user-facing changes?
Yes. Positive fixed-size binary fields retain their declared widths when
converting schemas in either direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
