Hi Neelesh,
> correct, nothing queries this today, and we have no proposal open against an OSS engine I'm not in favor of moving forward with the projection component until there is a OSS engine that will use it. As we've seen with variant it is complicated to get right and incorporate into engines. I feel like putting the effort into parquet at the moment would end up with bitrot/suboptimal design without concrete use. Hopefully we can make the concerns for documenting serialized values separate from the projection. Happy to take another look at the doc when you've updated it. Cheers, Micah On Friday, October 2, 2026, Neelesh Salian <[email protected]> wrote: > Hi Micah, > > Thanks for taking the time to review and giving your feedback here. > Responding: > > > 1. There are a lot of idiosyncrasies to getting the types > systems/structures lined up for any given serialization format to support > shredding (e.g. in protobuf, how to handle recursive schemas and well known > types can have multiple reasonable interpretations). > > Agreed, and we are aware of these. The type itself does no shredding and > interprets nothing; it keeps the bytes as the value, so recursion, > well-known types, and Any never have to be lined up with Parquet. For > projection, anything that does not map cleanly stays in the bytes and is > read by decoding rather than being forced into a column. We have this > working in a prototype and will lay out how each case is handled in the > write-up. > > > 2. If we expect parquet libraries to do the shredding, it means taking > on additional dependencies or maintaining parsing libraries or each > implementation. > > No, the Parquet library should have to parse. The type never decodes the > payload, so reading it is the bytes plus the two parameters. Shredding and > projection stay on the engine side, so the parser dependency lives in the > engine and not in any Parquet library. Originally I had this in the parquet > implementation locally, but I can minimize it in the POC and figure out the > right boundary for the engine. > > > 3. I don't think there are really any engines that would support > querying data in the form today. Do you have proposals open against any > OSS engines > to support this type? > > Correct, nothing queries this today, and we have no proposal open against > an OSS engine. The adjacent demand is real: engines are already building > projection and pushdown into encoded payload columns (Iceberg #16064 > <https://github.com/apache/iceberg/issues/16064>, DataFusion #21308 > <https://github.com/apache/datafusion/issues/21308>, Spark #58303 > <https://github.com/apache/spark/pull/58303>), all on Variant, which > re-encodes the bytes and cannot return the original message. That is the > gap this type fills where the exact bytes are the system of record, such as > audit, replay, lineage, and regulated data. We can prototype on one engine > and can bring it as a reference to open the discussion. > > > 4. More of a bikeshedding comment but "WIRE" might not be the best name > for > the type (suggested would be SERIALIZED_VALUE). > > Not attached to the name. Can go with SERIALIZED_VALUE since it is explicit > or open it up to the community for a different name. > > > IMO, I think a reasonable path forward might be to separate this into two > components: > > Yes. FORMAT as a free-form string with a recommended list (PROTO, THRIFT, > JSON, BSON). The schema identity is the part that needs the most thought, > and we will work it through in the write-up. > > > 2. Have a separate proposal on how to annotate projections on > SERIALIZED_VALUE columns in parquet. This would track a segment of the > schema as a projection of a SERIALIZED_VALUE column. This would allow > flexibility because we could use a variant shredding style for schemaless > serialization formats and the more natural structured shredding for values > with a schema. > > Agreed, that split is the right one: structured shredding for > schema-bearing formats and a Variant-shredding style for schemaless ones. > One of the reasons I kept it aligned with Variant shredding since it was > established. We can spell it out in a separate proposal. > > We will write up both proposals and update this thread. > > On Fri, Sep 25, 2026 at 2:17 PM Micah Kornfield <[email protected]> > wrote: > > > Hi Neelesh, > > Thanks for the proposal. While I think being able to allow annotating > > bytes or string columns as some sort of encoded structure I have a few > > concerns with the proposal: > > > > 1. There are a lot of idiosyncrasies to getting the types > > systems/structures lined up for any given serialization format to support > > shredding (e.g. in protobuf, how to handle recursive schemas and well > known > > types can have multiple reasonable interpretations). > > 2. If we expect parquet libraries to do the shredding, it means taking > on > > additional dependencies or maintaining parsing libraries or each > > implementation. > > 3. I don't think there are really any engines that would support > querying > > data in the form today. Do you have proposals open against any OSS > engines > > to support this type? > > 4. More of a bikeshedding comment but "WIRE" might not be the best name > > for the type (suggested would be SERIALIZED_VALUE). > > > > IMO, I think a reasonable path forward might be to separate this into two > > components: > > 1. Add a logical type that designates a column as a SERIALIZED_VALUE. > > This should probably hold two key elements: > > a. FORMAT (I think free-form string with standardized list is fine > for > > this. e.g. PROTO, THRIFT) > > b. SCHEMA: This needs some more thought but details on how the schema > > could be retrieved (whether stored inline or a registry, etc). Also if > it > > lacks a schema we should also capture this information (e.g. this could > > subsume existing JSON/BSON types today). > > 2. Have a separate proposal on how to annotate projections on > > SERIALIZED_VALUE columns in parquet. This would track a segment of the > > schema as a projection of a SERIALIZED_VALUE column. This would allow > > flexibility because we could use a variant shredding style for schemaless > > serialization formats and the more natural structured shredding for > values > > with a schema. > > > > Hopefully with these two components we could push the business logic of > the > > projection to engines leading to a more maintainable state. Before we > went > > too far with #2, I would like to see evidence that an engine would > actually > > support using the projections. > > > > Hope this helps. > > > > Thanks, > > -Micah > > > > On Fri, Sep 11, 2026 at 3:12 PM Neelesh Salian <[email protected] > > > > wrote: > > > > > Hi all, > > > > > > I have written up a proposal for a new Parquet logical type, > tentatively > > > called WIRE. > > > > > > The Proposal document: > > > > > > > > https://docs.google.com/document/d/1bt5YAeB0dvRT6hVt8bkw2AM3D5EyT > huyvtLxQBHpG5s/edit?usp=sharing > > > > > > > > > Serialized messages, protobuf especially, are a common way data is > stored > > > at rest. There are a few ways to put them in Parquet today, and each > one > > > costs you something: > > > > > > 1. Store the raw bytes as a blob. You cannot query anything inside > them. > > > 2. Explode every field into columns (what parquet-protobuf and > > > parquet-thrift do). You pay a full decode on every write, a migration > on > > > every schema change, > > > and you drop what does not fit a relational schema (unknown fields, > > oneof, > > > byte-exactness). > > > 3. Convert to Variant. You re-encode the message and carry a per-row > > > field-name dictionary that protobuf never needed. > > > > > > So today you either cannot query the data, or you pay a lot and lose > > > fidelity to make it queryable. > > > > > > WIRE keeps the message as it is and makes the fields you query > columnar. > > It > > > stores the message verbatim in a "value" column, so the original is > > > recoverable byte for byte and there is no re-encode or migration when > the > > > schema grows. > > > The fields you actually query are shredded into native Parquet columns > > with > > > real statistics and pushdown, reusing the Variant shredding layout. > > > There is no metadata column, because protobuf and thrift address fields > > by > > > number, so there is no name dictionary to carry, unlike Variant. > > > A reader that does not know WIRE just sees plain columns. > > > The first version covers protobuf and thrift compact, and the design is > > > encoding-general, so more can follow. > > > > > > I would appreciate feedback on two things: > > > > > > 1. The design, and whether this is worth bringing forward as a formal > > > proposal. If so, I can follow the proposal lifecycle and add it to the > > > Active Proposals list. > > > 2. What is needed for cross-implementation verification? With Variant, > it > > > came down to two implementations proven to read and write each other's > > > files against shared fixtures in parquet-testing, and I would like to > > plan > > > for that early. > > > > > > If there is interest, I plan to build the parquet-java reference > > > implementation and drive the spec work, and would help stand up the > > > cross-language verification. > > > > > > Looking forward to your thoughts. > > > > > > Thanks, > > > Neelesh > > > > > >
