puskarpeter opened a new pull request, #3750:
URL: https://github.com/apache/parquet-java/pull/3750
### Rationale for this change
Protobuf allows empty message definitions, but Parquet forbids empty groups.
Converting a message
that merely *contains* a field of an empty message type produces a schema
with an empty group,
which writer construction rejects with `InvalidSchemaException: Cannot write
a schema with an
empty group`. Such fields appear in real-world schemas (deprecated stubs,
marker/placeholder
messages), and a single one makes the whole message type unwritable.
### What changes are included in this PR?
`ProtoSchemaConverter.addMessageField` terminates a field whose message type
has no fields as a
`BINARY` column holding the serialized message — the same mechanism
PARQUET-1711 uses for
recursion beyond `maxRecursion`. Since an empty message serializes to zero
bytes, the column is
cheap, and field **presence** still round-trips (null = unset vs empty bytes
= set):
- singular field → `optional binary stub` (or `required`, per the field);
- repeated field, parquet-specs mode → LIST-wrapped binary via the existing
`addRepeatedPrimitive`, so element cardinality survives;
- repeated field, old style → `repeated binary`;
- map value type → `optional binary value` inside the `key_value` group
(keys stay typed).
`ProtoWriteSupport.createMessageWriter`'s existing truncated-field check
(primitive BINARY where a
message field was declared → `BinaryWriter`) is generalized to look through
the LIST/MAP wrapper
(`getGroupType` → `getContentType`), so the writer tree lines up with these
schemas.
A message that is empty at the **root** is still rejected — there is no
parent field to hold the
bytes, and a Parquet file with zero columns is not representable.
### Are these changes tested?
Yes. New `ProtoEmptyMessageTest` (new test messages `Stub`/`StubBox` in
`Trees.proto`) writes
through the real write path (`ProtoParquetWriter` → `MessageColumnIO`, both
specs-compliant and
old style) and reads back with `GroupReadSupport`:
- singular / repeated / map-value empty-message fields round-trip with
correct cardinality and
zero-byte values;
- presence round-trips (set empty message vs unset field);
- an empty root message still fails with `InvalidSchemaException` ("Cannot
write a schema with an
empty group").
`ProtoSchemaConverterTest.testEmptyMessageFields` pins the converted schema.
The full
parquet-protobuf suite passes (114 tests).
### Are there any user-facing changes?
Message types that previously could not be written to Parquet at all now
can; fields of empty
message types appear as (possibly LIST/MAP-wrapped) `binary` columns. No
change for schemas that
were previously writable. Error behavior for an empty root message is
unchanged.
Closes #2142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
