Oh interesting, so we could use this to annotate pretty much arbitrary data?


Twitter: https://twitter.com/holdenkarau
Fight Health Insurance: https://www.fighthealthinsurance.com/
<https://www.fighthealthinsurance.com/?q=hk_email>
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her

On Thu, Nov 12, 2026 at 1:21 PM Neelesh Salian <[email protected]>
wrote:

> Hi all,
>
> I've started a discussion on dev@parquet about a new logical type,
> SERIALIZED_VALUE [1][2], and I'd like
> the Spark community's feedback on whether Spark would support it.
>
> SERIALIZED_VALUE would annotate a BYTE_ARRAY column of serialized
> protobuf, FlatBuffers, or thrift
> messages. It would record the format of the bytes and store the schema in
> the file, along with the name of
> the root message type. For protobuf the schema would be a
> FileDescriptorSet. Parquet would never decode
> the bytes, and a reader that does not know the type would read the column
> as plain BYTE_ARRAY.
>
> Spark already decodes protobuf with *from_protobuf*, which takes a
> message name and a binary
> FileDescriptorSet, but the user has to supply both on every call. With
> both stored in the file, Spark
> could read them from there.
>
> Parquet would define the annotation and would store the bytes and the
> schema without parsing them, so it
> would take on no dependency on any serialization format. Spark would own
> the decoding and how fields are
> exposed to queries. A first step in Spark would be something small, like
> mapping the annotation to
> BinaryType and carrying the format and schema through on write. If there
> is agreement on the direction,
> I'd like to work with the community on an SPIP for reading message fields
> directly and for projected
> columns, which Parquet would take up once an engine wants them.
>
> Feedback on the direction is very welcome.
>
> Thanks,
> Neelesh
>
> [1] *Proposal:*
> https://docs.google.com/document/d/1RWH8P2PfMWMXvJuwtGXLe1HQbDjmL_gqR6wmhC7x3Ac/edit?usp=sharing
> [2] *Parquet discussion thread:*
> https://lists.apache.org/thread.html/5xn8o9rvrv1rsjs811t2gn0tkjq0zl02
>

Reply via email to