Hi all, I have written up a proposal for a new Parquet logical type, tentatively called WIRE.
The Proposal document: https://docs.google.com/document/d/1bt5YAeB0dvRT6hVt8bkw2AM3D5EyThuyvtLxQBHpG5s/edit?usp=sharing Serialized messages, protobuf especially, are a common way data is stored at rest. There are a few ways to put them in Parquet today, and each one costs you something: 1. Store the raw bytes as a blob. You cannot query anything inside them. 2. Explode every field into columns (what parquet-protobuf and parquet-thrift do). You pay a full decode on every write, a migration on every schema change, and you drop what does not fit a relational schema (unknown fields, oneof, byte-exactness). 3. Convert to Variant. You re-encode the message and carry a per-row field-name dictionary that protobuf never needed. So today you either cannot query the data, or you pay a lot and lose fidelity to make it queryable. WIRE keeps the message as it is and makes the fields you query columnar. It stores the message verbatim in a "value" column, so the original is recoverable byte for byte and there is no re-encode or migration when the schema grows. The fields you actually query are shredded into native Parquet columns with real statistics and pushdown, reusing the Variant shredding layout. There is no metadata column, because protobuf and thrift address fields by number, so there is no name dictionary to carry, unlike Variant. A reader that does not know WIRE just sees plain columns. The first version covers protobuf and thrift compact, and the design is encoding-general, so more can follow. I would appreciate feedback on two things: 1. The design, and whether this is worth bringing forward as a formal proposal. If so, I can follow the proposal lifecycle and add it to the Active Proposals list. 2. What is needed for cross-implementation verification? With Variant, it came down to two implementations proven to read and write each other's files against shared fixtures in parquet-testing, and I would like to plan for that early. If there is interest, I plan to build the parquet-java reference implementation and drive the spec work, and would help stand up the cross-language verification. Looking forward to your thoughts. Thanks, Neelesh
