Hi Holden, thanks for the comment. > Oh interesting, so we could use this to annotate pretty much arbitrary data? Not arbitrary data. For now it's scoped to serialized messages that have a schema, like protobuf, FlatBuffers, or thrift, and the file stores that schema alongside the column. The type is designed so more formats can be added later if there are use cases for them.
Did you have a particular kind of data in mind? On Thu, Nov 12, 2026 at 2:01 PM Holden Karau <[email protected]> wrote: > Oh interesting, so we could use this to annotate pretty much arbitrary > data? > > > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > On Thu, Nov 12, 2026 at 1:21 PM Neelesh Salian <[email protected]> > wrote: > >> Hi all, >> >> I've started a discussion on dev@parquet about a new logical type, >> SERIALIZED_VALUE [1][2], and I'd like >> the Spark community's feedback on whether Spark would support it. >> >> SERIALIZED_VALUE would annotate a BYTE_ARRAY column of serialized >> protobuf, FlatBuffers, or thrift >> messages. It would record the format of the bytes and store the schema in >> the file, along with the name of >> the root message type. For protobuf the schema would be a >> FileDescriptorSet. Parquet would never decode >> the bytes, and a reader that does not know the type would read the column >> as plain BYTE_ARRAY. >> >> Spark already decodes protobuf with *from_protobuf*, which takes a >> message name and a binary >> FileDescriptorSet, but the user has to supply both on every call. With >> both stored in the file, Spark >> could read them from there. >> >> Parquet would define the annotation and would store the bytes and the >> schema without parsing them, so it >> would take on no dependency on any serialization format. Spark would own >> the decoding and how fields are >> exposed to queries. A first step in Spark would be something small, like >> mapping the annotation to >> BinaryType and carrying the format and schema through on write. If there >> is agreement on the direction, >> I'd like to work with the community on an SPIP for reading message fields >> directly and for projected >> columns, which Parquet would take up once an engine wants them. >> >> Feedback on the direction is very welcome. >> >> Thanks, >> Neelesh >> >> [1] *Proposal:* >> https://docs.google.com/document/d/1RWH8P2PfMWMXvJuwtGXLe1HQbDjmL_gqR6wmhC7x3Ac/edit?usp=sharing >> [2] *Parquet discussion thread:* >> https://lists.apache.org/thread.html/5xn8o9rvrv1rsjs811t2gn0tkjq0zl02 >> >
