Hi all, I've started a discussion on dev@parquet about a new logical type, SERIALIZED_VALUE [1][2], and I'd like the Flink community's feedback on whether Flink would support it.
SERIALIZED_VALUE would annotate a BYTE_ARRAY column of serialized protobuf, FlatBuffers, or thrift messages. It would record the format of the bytes and store the schema in the file, along with the name of the root message type. For protobuf the schema would be a FileDescriptorSet. Parquet would never decode the bytes, and a reader that does not know the type would read the column as plain BYTE_ARRAY. >From my understanding, Flink's protobuf format needs the generated message class on the classpath, does not support the filesystem connector, and Javadoc suggests Parquet for storing protobuf messages in bulk. There are also open requests to read protobuf with a schema from the Confluent Schema Registry rather than a compiled class (FLINK-29731 <https://issues.apache.org/jira/browse/FLINK-29731>, FLINK-34440 <https://issues.apache.org/jira/browse/FLINK-34440>). A Parquet column that keeps the original bytes and carries its own schema would let Flink do the same for files. Parquet would define the annotation and would store the bytes and the schema without parsing them, so it would take on no dependency on any serialization format. Flink would own the decoding and how fields are exposed to queries. Reading the type would need parquet-java 1.18.0 or later, since the 1.15.2 version Flink uses fails on a logical type it does not recognize. If there is agreement on the direction, I'd like to work with the community on a FLIP for reading message fields directly and for projected columns, which Parquet would take up once an engine wants them. Feedback on the direction is very welcome. Thanks, Neelesh [1] *Proposal: * https://docs.google.com/document/d/1RWH8P2PfMWMXvJuwtGXLe1HQbDjmL_gqR6wmhC7x3Ac/edit?usp=sharing [2] *Parquet discussion thread:* https://lists.apache.org/thread.html/5xn8o9rvrv1rsjs811t2gn0tkjq0zl02
